Project

General

Profile

Bug #14088

mon: nothing logged when ENOSPC encountered during start up

Added by Tom Deneau almost 2 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Administration/Usability
Target version:
-
Start date:
12/15/2015
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
infernalis
Needs Doc:
No
Component(RADOS):
Monitor

Description

For example, ceph-mon process was dying and reporting only the
following to dmesg, nothing to /var/log/ceph/mon.log

init: ceph-mon (ceph/monhost) main process (16538) terminated with status 28
init: ceph-mon (ceph/monhost) main process ended, respawning
init: ceph-create-keys main process (16227) killed by TERM signal
init: ceph-mon (ceph/monhost) main process (16546) terminated with status 28
init: ceph-mon (ceph/monhost) main process ended, respawning
init: ceph-create-keys main process (16548) killed by TERM signal
init: ceph-mon (ceph/monhost) main process (16556) terminated with status 28
init: ceph-mon (ceph/monhost) main process ended, respawning
init: ceph-create-keys main process (16558) killed by TERM signal
init: ceph-mon (ceph/monhost) main process (16566) terminated with status 28
init: ceph-mon (ceph/monhost) respawning too fast, stopped
init: ceph-create-keys main process (16568) killed by TERM signal

It required looking at the sources to see that status 28 meant
that monhost's disk free space was very low.

From Brad Hubbard...

it may not have been anticipated that init would swallow these types of errors early in the process and just report the return code.

History

#1 Updated by Brad Hubbard almost 2 years ago

  • Category set to OSD
  • Assignee set to Brad Hubbard

#2 Updated by Brad Hubbard almost 2 years ago

  • Category changed from OSD to Monitor

#3 Updated by Josh Durgin 7 months ago

  • Subject changed from Fatal errors at daemon start-up are not logged to mon: nothing logged when ENOSPC encountered during start up

#4 Updated by Greg Farnum 5 months ago

  • Project changed from Ceph to RADOS
  • Category changed from Monitor to Administration/Usability
  • Component(RADOS) Monitor added

Brad, did you do any work on this?

#5 Updated by Brad Hubbard 5 months ago

Greg, No, but I can try and take a look in the next few days if you'd like?

#6 Updated by Greg Farnum 5 months ago

  • Assignee deleted (Brad Hubbard)

No, just scrubbing and trying to get things in a realistic state.

#7 Updated by Brad Hubbard 5 months ago

  • Status changed from New to In Progress
  • Assignee set to Brad Hubbard

#8 Updated by Brad Hubbard 5 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF