Bug #5255
0.56.6 -> cuttlefish tip (to be .3), mon crashes on boot
Status:
Resolved
Priority:
High
Assignee:
Joao Eduardo Luis
Category:
Monitor
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
Backport:
cuttlefish
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I upgraded my first mon from 0.56.6 to cuttlefish tip as of now (8544ea7) and it crashes on boot with:
root@ms-fe1001:~# /usr/bin/ceph-mon --cluster=ceph -i ms-fe1001 --debug-mon=20 --debug-ms=10 -f Invalid argument: /var/lib/ceph/mon/ceph-ms-fe1001/store.db: does not exist (create_if_missing is false) common/perf_counters.cc: In function 'void PerfCountersCollection::remove(PerfCounters*)' thread 7f4667213780 time 2013-06-05 17:03:00.883644 common/perf_counters.cc: 61: FAILED assert(i != m_loggers.end()) ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a) 1: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef] 2: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b] 3: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9] 4: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a] 5: (main()+0x44b) [0x48cb8b] 6: (__libc_start_main()+0xed) [0x7f46652c876d] 7: /usr/bin/ceph-mon() [0x49098d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2013-06-05 17:03:00.884895 7f4667213780 -1 common/perf_counters.cc: In function 'void PerfCountersCollection::remove(PerfCounters*)' thread 7f4667213780 time 2013-06-05 17:03:00.883644 common/perf_counters.cc: 61: FAILED assert(i != m_loggers.end()) ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a) 1: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef] 2: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b] 3: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9] 4: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a] 5: (main()+0x44b) [0x48cb8b] 6: (__libc_start_main()+0xed) [0x7f46652c876d] 7: /usr/bin/ceph-mon() [0x49098d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 0> 2013-06-05 17:03:00.884895 7f4667213780 -1 common/perf_counters.cc: In function 'void PerfCountersCollection::remove(PerfCounters*)' thread 7f4667213780 time 2013-06-05 17:03:00.883644 common/perf_counters.cc: 61: FAILED assert(i != m_loggers.end()) ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a) 1: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef] 2: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b] 3: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9] 4: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a] 5: (main()+0x44b) [0x48cb8b] 6: (__libc_start_main()+0xed) [0x7f46652c876d] 7: /usr/bin/ceph-mon() [0x49098d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (Aborted) ** in thread 7f4667213780 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a) 1: /usr/bin/ceph-mon() [0x5999fa] 2: (()+0xfcb0) [0x7f4666df2cb0] 3: (gsignal()+0x35) [0x7f46652dd425] 4: (abort()+0x17b) [0x7f46652e0b8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4665c2f69d] 6: (()+0xb5846) [0x7f4665c2d846] 7: (()+0xb5873) [0x7f4665c2d873] 8: (()+0xb596e) [0x7f4665c2d96e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6529ef] 10: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef] 11: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b] 12: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9] 13: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a] 14: (main()+0x44b) [0x48cb8b] 15: (__libc_start_main()+0xed) [0x7f46652c876d] 16: /usr/bin/ceph-mon() [0x49098d] 2013-06-05 17:03:00.886693 7f4667213780 -1 *** Caught signal (Aborted) ** in thread 7f4667213780 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a) 1: /usr/bin/ceph-mon() [0x5999fa] 2: (()+0xfcb0) [0x7f4666df2cb0] 3: (gsignal()+0x35) [0x7f46652dd425] 4: (abort()+0x17b) [0x7f46652e0b8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4665c2f69d] 6: (()+0xb5846) [0x7f4665c2d846] 7: (()+0xb5873) [0x7f4665c2d873] 8: (()+0xb596e) [0x7f4665c2d96e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6529ef] 10: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef] 11: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b] 12: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9] 13: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a] 14: (main()+0x44b) [0x48cb8b] 15: (__libc_start_main()+0xed) [0x7f46652c876d] 16: /usr/bin/ceph-mon() [0x49098d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 0> 2013-06-05 17:03:00.886693 7f4667213780 -1 *** Caught signal (Aborted) ** in thread 7f4667213780 ceph version 0.61.2-57-g8544ea7 (8544ea751884617616addc17b4467b9a86bd9d8a) 1: /usr/bin/ceph-mon() [0x5999fa] 2: (()+0xfcb0) [0x7f4666df2cb0] 3: (gsignal()+0x35) [0x7f46652dd425] 4: (abort()+0x17b) [0x7f46652e0b8b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f4665c2f69d] 6: (()+0xb5846) [0x7f4665c2d846] 7: (()+0xb5873) [0x7f4665c2d873] 8: (()+0xb596e) [0x7f4665c2d96e] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6529ef] 10: (PerfCountersCollection::remove(PerfCounters*)+0x24f) [0x6a04ef] 11: (LevelDBStore::~LevelDBStore()+0x1b) [0x58324b] 12: (LevelDBStore::~LevelDBStore()+0x9) [0x5834c9] 13: (Monitor::StoreConverter::needs_conversion()+0x20a) [0x4a950a] 14: (main()+0x44b) [0x48cb8b] 15: (__libc_start_main()+0xed) [0x7f46652c876d] 16: /usr/bin/ceph-mon() [0x49098d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Aborted
Associated revisions
os/LevelDBStore: only remove logger if non-null
Signed-off-by: Sage Weil <sage@inktank.com>
(cherry picked from commit ce67c58db7d3e259ef5a8222ef2ebb1febbf7362)
Fixes: #5255
History
#1 Updated by Ian Colle almost 11 years ago
- Assignee set to Joao Eduardo Luis
- Priority changed from High to Urgent
#2 Updated by Ian Colle almost 11 years ago
- Status changed from New to In Progress
#3 Updated by Joao Eduardo Luis almost 11 years ago
- Category set to Monitor
- Status changed from In Progress to Pending Backport
- Priority changed from Urgent to High
- Backport set to cuttlefish
Sage had already created a patch for this but it's only on next (ce67c58db7d3e259ef5a8222ef2ebb1febbf7362).
Backporting it to cuttlefish.
#4 Updated by Joao Eduardo Luis almost 11 years ago
backported with commit 7d549cb82ab8ebcf1cc104fc557d601b486c7635
#5 Updated by Faidon Liambotis almost 11 years ago
Just tested this and it works as expected.
#6 Updated by Joao Eduardo Luis almost 11 years ago
- Status changed from Pending Backport to Resolved