Bug #5292
closed
mon: monitor crashing due to not being in the monmap (no monmap to be in)
Added by Jeff Moskow almost 11 years ago.
Updated almost 11 years ago.
Description
I run a 4 node CEPH cluster (all are currently running 0.61.3 - upgraded to cuttlefish a few weeks ago) and (3 nodes in the mon quorom). This morning, one of the mons stopped and I can't restart it. The problem seems to be that it wants to re-create the keys using "/usr/sbin/ceph-create-keys i b" - which continually refused:
- /usr/sbin/ceph-create-keys -i b
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
Files
- Tracker changed from Support to Bug
- Assignee set to Joao Eduardo Luis
Can you share the monitor's logs with 'debug mon = 20' set?
I think that this is what you want, if not, just let me know.
Jeff
- Subject changed from Having problem with mon to mon: monitor crashing due to not being in the monmap (no monmap to be in)
- Category set to Monitor
- Status changed from New to In Progress
- Priority changed from High to Urgent
Monitor is not in the monmap because there is no monmap. This should be due to a sync bug (related to #5256) that removes the monmap backup after a failed sync -- it just takes the monitor to either crash or be killed/restarted for this to be triggered -- and it manifests itself later on due to the preforker bug. A fix for this should be hitting next tonight and will be backported to cuttlefish soon after.
- Status changed from In Progress to Resolved
Fix for this went into next and cuttlefish branches as of last night; see #5256.
I just tried apt-get update and it didn't pull down any cuttlefish updates. Have they been released? Do I need to do anything special to get/test them?
Thanks!
Jeff
I did a reboot, just to make sure :-(
- ceph -v
ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
- /usr/sbin/ceph-create-keys -i b
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
- Status changed from Resolved to Need More Info
Okay, can you post the monitor's logs with 'debug mon = 20' ?
Here you go. Please let me know if you need anything else.
Jeff
- Status changed from Need More Info to Resolved
You hit #5205 -- not the same issue, thus closing this ticket again.
Also available in: Atom
PDF