Project

General

Profile

Bug #13783

monitors crashing constantly with 0.94.5

Added by Tom Verdaat about 7 years ago. Updated about 7 years ago.

Status:
Duplicate
Priority:
Urgent
Category:
Monitor
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Deployed a new ceph hammer cluster on hardware using puppet. Last time we tested was with 0.94.3 which went fine, but this deployment with 0.94.5 does not work. The monitors keep crashing with a 'segmentation fault'. Already happened when we had one monitor. Now that we have three, usually two or all three go down. Seen this when processing crushmap, auth and osd commands so can't narrow down the cause.

Added logs of all three monitors with debug_mon level 20.

Could be a duplicate of issue #13748 but I can't tell so I created a separate ticket.

Definitely critical because this is making it impossible for us to run ceph!

ceph-mon.00219ba7d71d.log.tar.gz - ceph-mon.00219ba7d71d.log (481 KB) Tom Verdaat, 11/12/2015 06:09 PM

ceph-mon.f04da200e0ee.log.tar.gz - ceph-mon.f04da200e0ee.log (683 KB) Tom Verdaat, 11/12/2015 06:09 PM

ceph-mon.f04da200dc1d.log.1.tar.gz - ceph-mon.f04da200dc1d.log part 1 (694 KB) Tom Verdaat, 11/12/2015 06:16 PM

ceph-mon.f04da200dc1d.log.2.tar.gz - ceph-mon.f04da200dc1d.log part 2 (769 KB) Tom Verdaat, 11/12/2015 06:16 PM


Related issues

Related to Ceph - Bug #13748: ceph-mons crashing constantly after 0.94.3->0.94.5 upgrade Resolved 11/10/2015

History

#1 Updated by Tom Verdaat about 7 years ago

Logs

#2 Updated by Tom Verdaat about 7 years ago

Logs

#3 Updated by Tom Verdaat about 7 years ago

Logs now small enough to be attached :)

#4 Updated by Joao Eduardo Luis about 7 years ago

Yep, looks like a duplicate. Let me make the same request I made on the other ticket: can you share your monitor's store.db?

#5 Updated by Joao Eduardo Luis about 7 years ago

  • Assignee set to Joao Eduardo Luis

#6 Updated by Tom Verdaat about 7 years ago

Done! They were slightly large so I sent them to you by e-mail.

#7 Updated by Joao Eduardo Luis about 7 years ago

got it. thanks!

#8 Updated by Nathan Cutler about 7 years ago

  • Related to Bug #13748: ceph-mons crashing constantly after 0.94.3->0.94.5 upgrade added

#9 Updated by Joao Eduardo Luis about 7 years ago

  • Category set to Monitor
  • Priority changed from Normal to Urgent

#10 Updated by Tom Verdaat about 7 years ago

Quick note: we've upgraded to infernalis and it runs as expected, as also confirmed by Logan V in the other ticket. Means this regression is limited to hammer!

#11 Updated by Joao Eduardo Luis about 7 years ago

  • Status changed from New to Duplicate

From the trace I can tell this is indeed a duplicate of #13748. Will follow up there.

Also available in: Atom PDF