Project

General

Profile

Bug #5255

0.56.6 -> cuttlefish tip (to be .3), mon crashes on boot

Added by Faidon Liambotis over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
High
Assignee:
Joao Eduardo Luis
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
cuttlefish
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I upgraded my first mon from 0.56.6 to cuttlefish tip as of now (8544ea7) and it crashes on boot with:

root@ms-fe1001:~# /usr/bin/ceph-mon --cluster=ceph -i ms-fe1001 --debug-mon=20 --debug-ms=10 -f
Invalid argument: /var/lib/ceph/mon/ceph-ms-fe1001/store.db: does not exist (create_if_missing is false)
common/perf_counters.cc: In function 'void PerfCountersCollection::remove(PerfCounters*)' thread 7f4667213780 time 2013-06-05 17:03:00.883644
common/perf_counters.cc: 61: FAILED assert(i != m_loggers.end())
 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a)
 1: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef]
 2: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b]
 3: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9]
 4: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a]
 5: (main()+0x44b) [0x48cb8b]
 6: (__libc_start_main()+0xed) [0x7f46652c876d]
 7: /usr/bin/ceph-mon() [0x49098d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2013-06-05 17:03:00.884895 7f4667213780 -1 common/perf_counters.cc: In function 'void PerfCountersCollection::remove(PerfCounters*)' thread 7f4667213780 time 2013-06-05 17:03:00.883644
common/perf_counters.cc: 61: FAILED assert(i != m_loggers.end())

 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a)
 1: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef]
 2: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b]
 3: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9]
 4: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a]
 5: (main()+0x44b) [0x48cb8b]
 6: (__libc_start_main()+0xed) [0x7f46652c876d]
 7: /usr/bin/ceph-mon() [0x49098d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

     0> 2013-06-05 17:03:00.884895 7f4667213780 -1 common/perf_counters.cc: In function 'void PerfCountersCollection::remove(PerfCounters*)' thread 7f4667213780 time 2013-06-05 17:03:00.883644
common/perf_counters.cc: 61: FAILED assert(i != m_loggers.end())

 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a)
 1: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef]
 2: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b]
 3: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9]
 4: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a]
 5: (main()+0x44b) [0x48cb8b]
 6: (__libc_start_main()+0xed) [0x7f46652c876d]
 7: /usr/bin/ceph-mon() [0x49098d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

terminate called after throwing an instance of 'ceph::FailedAssertion'
*** Caught signal (Aborted) **
 in thread 7f4667213780
 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a)
 1: /usr/bin/ceph-mon() [0x5999fa]
 2: (()+0xfcb0) [0x7f4666df2cb0]
 3: (gsignal()+0x35) [0x7f46652dd425]
 4: (abort()+0x17b) [0x7f46652e0b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4665c2f69d]
 6: (()+0xb5846) [0x7f4665c2d846]
 7: (()+0xb5873) [0x7f4665c2d873]
 8: (()+0xb596e) [0x7f4665c2d96e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6529ef]
 10: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef]
 11: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b]
 12: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9]
 13: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a]
 14: (main()+0x44b) [0x48cb8b]
 15: (__libc_start_main()+0xed) [0x7f46652c876d]
 16: /usr/bin/ceph-mon() [0x49098d]
2013-06-05 17:03:00.886693 7f4667213780 -1 *** Caught signal (Aborted) **
 in thread 7f4667213780

 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a)
 1: /usr/bin/ceph-mon() [0x5999fa]
 2: (()+0xfcb0) [0x7f4666df2cb0]
 3: (gsignal()+0x35) [0x7f46652dd425]
 4: (abort()+0x17b) [0x7f46652e0b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4665c2f69d]
 6: (()+0xb5846) [0x7f4665c2d846]
 7: (()+0xb5873) [0x7f4665c2d873]
 8: (()+0xb596e) [0x7f4665c2d96e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6529ef]
 10: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef]
 11: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b]
 12: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9]
 13: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a]
 14: (main()+0x44b) [0x48cb8b]
 15: (__libc_start_main()+0xed) [0x7f46652c876d]
 16: /usr/bin/ceph-mon() [0x49098d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

     0> 2013-06-05 17:03:00.886693 7f4667213780 -1 *** Caught signal (Aborted) **
 in thread 7f4667213780

 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a)
 1: /usr/bin/ceph-mon() [0x5999fa]
 2: (()+0xfcb0) [0x7f4666df2cb0]
 3: (gsignal()+0x35) [0x7f46652dd425]
 4: (abort()+0x17b) [0x7f46652e0b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4665c2f69d]
 6: (()+0xb5846) [0x7f4665c2d846]
 7: (()+0xb5873) [0x7f4665c2d873]
 8: (()+0xb596e) [0x7f4665c2d96e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6529ef]
 10: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef]
 11: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b]
 12: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9]
 13: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a]
 14: (main()+0x44b) [0x48cb8b]
 15: (__libc_start_main()+0xed) [0x7f46652c876d]
 16: /usr/bin/ceph-mon() [0x49098d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Aborted

Associated revisions

Revision 7d549cb8 (diff)
Added by Sage Weil over 10 years ago

os/LevelDBStore: only remove logger if non-null

Signed-off-by: Sage Weil <>
(cherry picked from commit ce67c58db7d3e259ef5a8222ef2ebb1febbf7362)
Fixes: #5255

History

#1 Updated by Ian Colle over 10 years ago

  • Assignee set to Joao Eduardo Luis
  • Priority changed from High to Urgent

#2 Updated by Ian Colle over 10 years ago

  • Status changed from New to In Progress

#3 Updated by Joao Eduardo Luis over 10 years ago

  • Category set to Monitor
  • Status changed from In Progress to Pending Backport
  • Priority changed from Urgent to High
  • Backport set to cuttlefish

Sage had already created a patch for this but it's only on next (ce67c58db7d3e259ef5a8222ef2ebb1febbf7362).

Backporting it to cuttlefish.

#4 Updated by Joao Eduardo Luis over 10 years ago

#5 Updated by Faidon Liambotis over 10 years ago

Just tested this and it works as expected.

#6 Updated by Joao Eduardo Luis over 10 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF