Bug #1414
closedcmon crash Paxos::handle_collect (this=0x22a9000, collect=0x22e56c0) at ../../src/mon/Paxos.cc:145
Description
I see this repeatedly in my current setup:
Core was generated by `/usr/ceph/bin/cmon i bravo -c /etc/ceph/ceph.conf'.
Program terminated with signal 6, Aborted.
#0 0x00007fd9e6af0b3b in raise ()---Type <return> to continue, or q <return> to quit--
(gdb) bt
#0 0x00007fd9e6af0b3b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000007b4c16 in reraise_fatal (signum=6) at ../../src/global/signal_handler.cc:59
#2 0x00000000007b4e34 in handle_fatal_signal (signum=6) at ../../src/global/signal_handler.cc:106
#3 <signal handler called>
#4 0x00007fd9e547ad05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fd9e547eab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007fd9e5d316dd in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fd9e5d2f926 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007fd9e5d2f953 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007fd9e5d2fa5e in _cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00000000007b48bb in ceph::_ceph_assert_fail (assertion=0x7cf8b7 "bl.length() > 0", file=0x7cf7f8 "../../src/mon/Paxos.cc", line=145, func=0x7d0880 "void Paxos::handle_collect(MMonPaxos*)") at ../../src/common/assert.cc:70
#11 0x000000000064ed05 in Paxos::handle_collect (this=0x22a9000, collect=0x22e56c0) at ../../src/mon/Paxos.cc:145
#12 0x00000000006555b4 in Paxos::dispatch (this=0x22a9000, m=0x22e56c0) at ../../src/mon/Paxos.cc:820
#13 0x000000000062fd9d in Monitor::_ms_dispatch (this=0x22cfa00, m=0x22e56c0) at ../../src/mon/Monitor.cc:800
#14 0x000000000063726c in Monitor::ms_dispatch (this=0x22cfa00, m=0x22e56c0) at ../../src/mon/Monitor.h:204
#15 0x0000000000780622 in Messenger::ms_deliver_dispatch (this=0x22cf000, m=0x22e56c0) at ../../src/msg/Messenger.h:102
#16 0x000000000076a264 in SimpleMessenger::dispatch_entry (this=0x22cf000) at ../../src/msg/SimpleMessenger.cc:356
#17 0x000000000061c8ae in SimpleMessenger::DispatchThread::entry (this=0x22cf488) at ../../src/msg/SimpleMessenger.h:546
#18 0x00000000006f07d5 in Thread::_entry_func (arg=0x22cf488) at ../../src/common/Thread.cc:45
#19 0x00007fd9e6ae7d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007fd9e552d04d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()
The last part of the log is:
2011-08-19 00:13:54.231567 7f32e639a760 log [INF] : mon.bravo calling new monitor election2011-08-19 00:13:54.243782 7f32e6396700 -- 192.168.101.113:6789/0 >> 192.168.101.112:6789/0 pipe(0x2d47500 sd=9 pgs=0 cs=0 l=0).fault first fault
2011-08-19 00:13:54.243874 7f32e16b0700 -- 192.168.101.113:6789/0 >> 192.168.101.115:6789/0 pipe(0x2d47280 sd=10 pgs=0 cs=0 l=0).fault first fault
2011-08-19 00:13:54.282302 7f32e15af700 -- 192.168.101.113:6789/0 >> 192.168.101.112:6789/0 pipe(0x2d47780 sd=11 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
2011-08-19 00:13:54.397402 7f32e12ac700 -- 192.168.101.113:6789/0 >> 192.168.101.115:6789/0 pipe(0x2d47500 sd=12 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
../../src/mon/Paxos.cc: In function 'void Paxos::handle_collect(MMonPaxos*)', in thread '0x7f32e26b2700'
../../src/mon/Paxos.cc: 145: FAILED assert(bl.length() > 0)
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x7b3261]
2: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64e945]
3: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6551f4]
4: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62f9dd]
5: (Monitor::ms_dispatch(Message*)+0x3a) [0x636eac]
6: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x77f13a]
7: (SimpleMessenger::dispatch_entry()+0x810) [0x768d74]
8: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c4ee]
9: (Thread::_entry_func(void*)+0x23) [0x6f03cd]
10: (()+0x6d8c) [0x7f32e5f70d8c]
11: (clone()+0x6d) [0x7f32e49b604d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x7b3261]
2: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64e945]
3: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6551f4]
4: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62f9dd]
5: (Monitor::ms_dispatch(Message*)+0x3a) [0x636eac]
6: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x77f13a]
7: (SimpleMessenger::dispatch_entry()+0x810) [0x768d74]
8: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c4ee]
9: (Thread::_entry_func(void*)+0x23) [0x6f03cd]
10: (()+0x6d8c) [0x7f32e5f70d8c]
11: (clone()+0x6d) [0x7f32e49b604d]
- Caught signal (Aborted) *
in thread 0x7f32e26b2700
ceph version (commit:)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x799f05]
2: /usr/ceph/bin/cmon() [0x7b38d3]
3: (()+0xfc60) [0x7f32e5f79c60]
4: (gsignal()+0x35) [0x7f32e4903d05]
5: (abort()+0x186) [0x7f32e4907ab6]
6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f32e51ba6dd]
7: (()+0xb9926) [0x7f32e51b8926]
8: (()+0xb9953) [0x7f32e51b8953]
9: (()+0xb9a5e) [0x7f32e51b8a5e]
10: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1f3) [0x7b33cb]
11: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64e945]
12: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6551f4]
13: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62f9dd]
14: (Monitor::ms_dispatch(Message*)+0x3a) [0x636eac]
15: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x77f13a]
16: (SimpleMessenger::dispatch_entry()+0x810) [0x768d74]
17: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c4ee]
18: (Thread::_entry_func(void*)+0x23) [0x6f03cd]
19: (()+0x6d8c) [0x7f32e5f70d8c]
20: (clone()+0x6d) [0x7f32e49b604d]
2011-08-19 09:48:04.467717 7fd9e6f11760 ceph version .commit: . process: cmon. pid: 9320
2011-08-19 09:48:04.468163 7fd9e6f11760 store(/data/mon.bravo) mount
2011-08-19 09:48:04.470538 7fd9e6f11760 mon.bravo@1(starting) e1 init fsid 12b23328-79c8-09a0-6be5-d154d246232b
2011-08-19 09:48:04.478382 7fd9e6f11760 log [INF] : mon.bravo calling new monitor election
2011-08-19 09:48:04.489760 7fd9e2227700 -- 192.168.101.113:6789/0 >> 192.168.101.115:6789/0 pipe(0x22d0280 sd=10 pgs=0 cs=0 l=0).fault first fault
2011-08-19 09:48:04.569612 7fd9e1f24700 -- 192.168.101.113:6789/0 >> 192.168.101.115:6789/0 pipe(0x22d0c80 sd=12 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
2011-08-19 09:48:04.640694 7fd9e1e23700 -- 192.168.101.113:6789/0 >> 192.168.101.112:6789/0 pipe(0x22d0a00 sd=13 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 6
../../src/mon/Paxos.cc: In function 'void Paxos::handle_collect(MMonPaxos*)', in thread '0x7fd9e3229700'
../../src/mon/Paxos.cc: 145: FAILED assert(bl.length() > 0)
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x7b4751]
2: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64ed05]
3: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6555b4]
4: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62fd9d]
5: (Monitor::ms_dispatch(Message*)+0x3a) [0x63726c]
6: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x780622]
7: (SimpleMessenger::dispatch_entry()+0x810) [0x76a264]
8: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c8ae]
9: (Thread::_entry_func(void*)+0x23) [0x6f07d5]
10: (()+0x6d8c) [0x7fd9e6ae7d8c]
11: (clone()+0x6d) [0x7fd9e552d04d]
ceph version (commit:)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x7b4751]
2: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64ed05]
3: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6555b4]
4: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62fd9d]
5: (Monitor::ms_dispatch(Message*)+0x3a) [0x63726c]
6: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x780622]
7: (SimpleMessenger::dispatch_entry()+0x810) [0x76a264]
8: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c8ae]
9: (Thread::_entry_func(void*)+0x23) [0x6f07d5]
10: (()+0x6d8c) [0x7fd9e6ae7d8c]
11: (clone()+0x6d) [0x7fd9e552d04d] - Caught signal (Aborted) *
in thread 0x7fd9e3229700
ceph version (commit:)
1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x79b3ed]
2: /usr/ceph/bin/cmon() [0x7b4dc3]
3: (()+0xfc60) [0x7fd9e6af0c60]
4: (gsignal()+0x35) [0x7fd9e547ad05]
5: (abort()+0x186) [0x7fd9e547eab6]
6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7fd9e5d316dd]
7: (()+0xb9926) [0x7fd9e5d2f926]
8: (()+0xb9953) [0x7fd9e5d2f953]
9: (()+0xb9a5e) [0x7fd9e5d2fa5e]
10: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1f3) [0x7b48bb]
11: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64ed05]
12: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6555b4]
13: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62fd9d]
14: (Monitor::ms_dispatch(Message*)+0x3a) [0x63726c]
15: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x780622]
16: (SimpleMessenger::dispatch_entry()+0x810) [0x76a264]
17: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c8ae]
18: (Thread::_entry_func(void*)+0x23) [0x6f07d5]
19: (()+0x6d8c) [0x7fd9e6ae7d8c]
20: (clone()+0x6d) [0x7fd9e552d04d]