Project

General

Profile

Actions
Copy link

Bug #5292

closed

mon: monitor crashing due to not being in the monmap (no monmap to be in)

Added by Jeff Moskow almost 11 years ago. Updated almost 11 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Joao Eduardo Luis
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I run a 4 node CEPH cluster (all are currently running 0.61.3 - upgraded to cuttlefish a few weeks ago) and (3 nodes in the mon quorom). This morning, one of the mons stopped and I can't restart it. The problem seems to be that it wants to re-create the keys using "/usr/sbin/ceph-create-keys i b" - which continually refused:

  1. /usr/sbin/ceph-create-keys -i b
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.

Files

Download all files
ceph-mon.b.log (22.5 KB) ceph-mon.b.log Jeff Moskow, 06/11/2013 08:34 AM
ceph-mon.b.log (7.58 KB) ceph-mon.b.log Jeff Moskow, 06/20/2013 06:48 AM

Related issues 1 (0 open — 1 closed)

Related to Ceph - Bug #5256: Upgraded bobtail->cuttlefish mon crashes, then can't resume the conversionResolvedJoao Eduardo Luis06/05/2013

Actions
Actions
Copy link
 #1

Updated by Ian Colle almost 11 years ago

  • Tracker changed from Support to Bug
  • Assignee set to Joao Eduardo Luis
Actions
Copy link
 #2

Updated by Joao Eduardo Luis almost 11 years ago

Can you share the monitor's logs with 'debug mon = 20' set?

Actions
Copy link
 #3

Updated by Jeff Moskow almost 11 years ago

I think that this is what you want, if not, just let me know.

Jeff

Actions
Copy link
 #4

Updated by Joao Eduardo Luis almost 11 years ago

  • Subject changed from Having problem with mon to mon: monitor crashing due to not being in the monmap (no monmap to be in)
  • Category set to Monitor
  • Status changed from New to In Progress
  • Priority changed from High to Urgent

Monitor is not in the monmap because there is no monmap. This should be due to a sync bug (related to #5256) that removes the monmap backup after a failed sync -- it just takes the monitor to either crash or be killed/restarted for this to be triggered -- and it manifests itself later on due to the preforker bug. A fix for this should be hitting next tonight and will be backported to cuttlefish soon after.

Actions
Copy link
 #5

Updated by Joao Eduardo Luis almost 11 years ago

  • Status changed from In Progress to Resolved

Fix for this went into next and cuttlefish branches as of last night; see #5256.

Actions
Copy link
 #6

Updated by Jeff Moskow almost 11 years ago

I just tried apt-get update and it didn't pull down any cuttlefish updates. Have they been released? Do I need to do anything special to get/test them?

Thanks!

Jeff

Actions
Copy link
 #7

Updated by Joao Eduardo Luis almost 11 years ago

It was backported to the branch that should soon become 0.61.4. Until then, you'll be able to find it on the gitbuilders.

http://ceph.com/docs/next/install/debian/#development-testing-packages

and take your pick from here:

http://gitbuilder.ceph.com/

Let us know how it goes.

Actions
Copy link
 #8

Updated by Jeff Moskow almost 11 years ago

I did a reboot, just to make sure :-(

  1. ceph -v
    ceph version 0.61.4 (1669132fcfc27d0c0b5e5bb93ade59d147e23404)
  1. /usr/sbin/ceph-create-keys -i b
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
    connect to /var/run/ceph/ceph-mon.b.asok failed with (111) Connection refused
    INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
Actions
Copy link
 #9

Updated by Joao Eduardo Luis almost 11 years ago

  • Status changed from Resolved to Need More Info

Okay, can you post the monitor's logs with 'debug mon = 20' ?

Actions
Copy link
 #10

Updated by Jeff Moskow almost 11 years ago

Here you go. Please let me know if you need anything else.

Jeff

Actions
Copy link
 #11

Updated by Joao Eduardo Luis almost 11 years ago

  • Status changed from Need More Info to Resolved

You hit #5205 -- not the same issue, thus closing this ticket again.

Actions
Copy link
 #12

Updated by Jeff Moskow almost 11 years ago

Thanks.

Actions
Copy link

Also available in: Atom PDF