Project

General

Profile

Bug #19291

mds: log rotation doesn't work if mds has respawned

Added by Patrick Donnelly 8 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Urgent
Category:
-
Target version:
-
Start date:
03/16/2017
Due date:
% Done:

0%

Source:
Development
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Component(FS):
MDS
Needs Doc:
No

Description

If an MDS respawns then its "comm" name becomes "exe" which confuses logrotate since it relies on killlall. What ends up happening is that logrotate will rename the current log to e.g. "ceph-mds.li1015-93.log.1", send SIGHUP to all processes named "ceph-mds" (there are none!), gzip the ceph-mds.li1015-93.log.1 to ceph-mds.li1015-93.log.1.gz and then unlink the original log ceph-mds.li1015-93.log.1. Unfortunately because ceph-mds does not get the SIGHUP, it will keep writing to the unlinked log inode until the disk space is consumed or ceph-mds dies.

This problem was introduced in 66a122025f6cf023cf7b2f3d8fbe4964fb7568a7 and discussed on ceph-devel here: https://www.spinics.net/lists/ceph-devel/msg33780.html

I'm going to suggest the simple (but very slightly racy) solution of calling prctl on startup to unconditionally change the name of the ceph-mds daemon to "ceph-mds". Thoughts?


Related issues

Copied to fs - Backport #19466: jewel: mds: log rotation doesn't work if mds has respawned Resolved

History

#1 Updated by Patrick Donnelly 8 months ago

  • Assignee set to Patrick Donnelly
  • Priority changed from Normal to Urgent
  • Source set to Development
  • Component(FS) MDS added

#2 Updated by Patrick Donnelly 8 months ago

Sorry hit submit on accident before finishing writing this up. Standby!

#3 Updated by Patrick Donnelly 8 months ago

  • Description updated (diff)

#4 Updated by Patrick Donnelly 8 months ago

  • Status changed from New to Need Review

#5 Updated by John Spray 8 months ago

  • Status changed from Need Review to Resolved

#6 Updated by Patrick Donnelly 8 months ago

  • Status changed from Resolved to Pending Backport

The bug is also in jewel 10.2.6, due to 6efad699249ba7c6928193dba111dbb23b606beb.

#7 Updated by Nathan Cutler 8 months ago

  • Backport set to jewel

#8 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #19466: jewel: mds: log rotation doesn't work if mds has respawned added

#9 Updated by Nathan Cutler 3 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF