Project

General

Profile

Actions
Copy link

Bug #19291

closed

mds: log rotation doesn't work if mds has respawned

Added by Patrick Donnelly about 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Patrick Donnelly
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

If an MDS respawns then its "comm" name becomes "exe" which confuses logrotate since it relies on killlall. What ends up happening is that logrotate will rename the current log to e.g. "ceph-mds.li1015-93.log.1", send SIGHUP to all processes named "ceph-mds" (there are none!), gzip the ceph-mds.li1015-93.log.1 to ceph-mds.li1015-93.log.1.gz and then unlink the original log ceph-mds.li1015-93.log.1. Unfortunately because ceph-mds does not get the SIGHUP, it will keep writing to the unlinked log inode until the disk space is consumed or ceph-mds dies.

This problem was introduced in 66a122025f6cf023cf7b2f3d8fbe4964fb7568a7 and discussed on ceph-devel here: https://www.spinics.net/lists/ceph-devel/msg33780.html

I'm going to suggest the simple (but very slightly racy) solution of calling prctl on startup to unconditionally change the name of the ceph-mds daemon to "ceph-mds". Thoughts?

Related issues 1 (0 open — 1 closed)

Copied to CephFS - Backport #19466: jewel: mds: log rotation doesn't work if mds has respawnedResolvedNathan CutlerActions
Actions
Copy link

Also available in: Atom PDF