Bug #19291: mds: log rotation doesn't work if mds has respawned - CephFS - Ceph

Actions

Copy link

Bug #19291

closed

mds: log rotation doesn't work if mds has respawned

Added by Patrick Donnelly about 7 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Patrick Donnelly

Category:

Target version:

% Done:

Source:

Development

Tags:

Backport:

jewel

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

MDS

Labels (FS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

If an MDS respawns then its "comm" name becomes "exe" which confuses logrotate since it relies on killlall. What ends up happening is that logrotate will rename the current log to e.g. "ceph-mds.li1015-93.log.1", send SIGHUP to all processes named "ceph-mds" (there are none!), gzip the ceph-mds.li1015-93.log.1 to ceph-mds.li1015-93.log.1.gz and then unlink the original log ceph-mds.li1015-93.log.1. Unfortunately because ceph-mds does not get the SIGHUP, it will keep writing to the unlinked log inode until the disk space is consumed or ceph-mds dies.

This problem was introduced in 66a122025f6cf023cf7b2f3d8fbe4964fb7568a7 and discussed on ceph-devel here: https://www.spinics.net/lists/ceph-devel/msg33780.html

I'm going to suggest the simple (but very slightly racy) solution of calling prctl on startup to unconditionally change the name of the ceph-mds daemon to "ceph-mds". Thoughts?

Related issues 1 (0 open — 1 closed)