Ceph

I see this repeatedly in my current setup:

Core was generated by `/usr/ceph/bin/cmon i bravo -c /etc/ceph/ceph.conf'.
Program terminated with signal 6, Aborted.
#0 0x00007fd9e6af0b3b in raise ()---Type <return> to continue, or q <return> to quit--
(gdb) bt
#0 0x00007fd9e6af0b3b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000007b4c16 in reraise_fatal (signum=6) at ../../src/global/signal_handler.cc:59
#2 0x00000000007b4e34 in handle_fatal_signal (signum=6) at ../../src/global/signal_handler.cc:106
#3 <signal handler called>
#4 0x00007fd9e547ad05 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fd9e547eab6 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007fd9e5d316dd in _gnu_cxx::_verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007fd9e5d2f926 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00007fd9e5d2f953 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9 0x00007fd9e5d2fa5e in _cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00000000007b48bb in ceph::_ceph_assert_fail (assertion=0x7cf8b7 "bl.length() > 0", file=0x7cf7f8 "../../src/mon/Paxos.cc", line=145, func=0x7d0880 "void Paxos::handle_collect(MMonPaxos*)") at ../../src/common/assert.cc:70
#11 0x000000000064ed05 in Paxos::handle_collect (this=0x22a9000, collect=0x22e56c0) at ../../src/mon/Paxos.cc:145
#12 0x00000000006555b4 in Paxos::dispatch (this=0x22a9000, m=0x22e56c0) at ../../src/mon/Paxos.cc:820
#13 0x000000000062fd9d in Monitor::_ms_dispatch (this=0x22cfa00, m=0x22e56c0) at ../../src/mon/Monitor.cc:800
#14 0x000000000063726c in Monitor::ms_dispatch (this=0x22cfa00, m=0x22e56c0) at ../../src/mon/Monitor.h:204
#15 0x0000000000780622 in Messenger::ms_deliver_dispatch (this=0x22cf000, m=0x22e56c0) at ../../src/msg/Messenger.h:102
#16 0x000000000076a264 in SimpleMessenger::dispatch_entry (this=0x22cf000) at ../../src/msg/SimpleMessenger.cc:356
#17 0x000000000061c8ae in SimpleMessenger::DispatchThread::entry (this=0x22cf488) at ../../src/msg/SimpleMessenger.h:546
#18 0x00000000006f07d5 in Thread::_entry_func (arg=0x22cf488) at ../../src/common/Thread.cc:45
#19 0x00007fd9e6ae7d8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#20 0x00007fd9e552d04d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x0000000000000000 in ?? ()

The last part of the log is:

• Caught signal (Aborted) *
  in thread 0x7f32e26b2700
  ceph version (commit:)
  1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x799f05]
  2: /usr/ceph/bin/cmon() [0x7b38d3]
  3: (()+0xfc60) [0x7f32e5f79c60]
  4: (gsignal()+0x35) [0x7f32e4903d05]
  5: (abort()+0x186) [0x7f32e4907ab6]
  6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f32e51ba6dd]
  7: (()+0xb9926) [0x7f32e51b8926]
  8: (()+0xb9953) [0x7f32e51b8953]
  9: (()+0xb9a5e) [0x7f32e51b8a5e]
  10: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1f3) [0x7b33cb]
  11: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64e945]
  12: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6551f4]
  13: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62f9dd]
  14: (Monitor::ms_dispatch(Message*)+0x3a) [0x636eac]
  15: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x77f13a]
  16: (SimpleMessenger::dispatch_entry()+0x810) [0x768d74]
  17: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c4ee]
  18: (Thread::_entry_func(void*)+0x23) [0x6f03cd]
  19: (()+0x6d8c) [0x7f32e5f70d8c]
  20: (clone()+0x6d) [0x7f32e49b604d]
  2011-08-19 09:48:04.467717 7fd9e6f11760 ceph version .commit: . process: cmon. pid: 9320
  2011-08-19 09:48:04.468163 7fd9e6f11760 store(/data/mon.bravo) mount
  2011-08-19 09:48:04.470538 7fd9e6f11760 mon.bravo@1(starting) e1 init fsid 12b23328-79c8-09a0-6be5-d154d246232b
  2011-08-19 09:48:04.478382 7fd9e6f11760 log [INF] : mon.bravo calling new monitor election
  2011-08-19 09:48:04.489760 7fd9e2227700 -- 192.168.101.113:6789/0 >> 192.168.101.115:6789/0 pipe(0x22d0280 sd=10 pgs=0 cs=0 l=0).fault first fault
  2011-08-19 09:48:04.569612 7fd9e1f24700 -- 192.168.101.113:6789/0 >> 192.168.101.115:6789/0 pipe(0x22d0c80 sd=12 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 1
  2011-08-19 09:48:04.640694 7fd9e1e23700 -- 192.168.101.113:6789/0 >> 192.168.101.112:6789/0 pipe(0x22d0a00 sd=13 pgs=0 cs=0 l=0).accept connect_seq 0 vs existing 0 state 6
  ../../src/mon/Paxos.cc: In function 'void Paxos::handle_collect(MMonPaxos*)', in thread '0x7fd9e3229700'
  ../../src/mon/Paxos.cc: 145: FAILED assert(bl.length() > 0)
  ceph version (commit:)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x7b4751]
  2: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64ed05]
  3: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6555b4]
  4: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62fd9d]
  5: (Monitor::ms_dispatch(Message*)+0x3a) [0x63726c]
  6: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x780622]
  7: (SimpleMessenger::dispatch_entry()+0x810) [0x76a264]
  8: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c8ae]
  9: (Thread::_entry_func(void*)+0x23) [0x6f07d5]
  10: (()+0x6d8c) [0x7fd9e6ae7d8c]
  11: (clone()+0x6d) [0x7fd9e552d04d]
  ceph version (commit:)
  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x89) [0x7b4751]
  2: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64ed05]
  3: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6555b4]
  4: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62fd9d]
  5: (Monitor::ms_dispatch(Message*)+0x3a) [0x63726c]
  6: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x780622]
  7: (SimpleMessenger::dispatch_entry()+0x810) [0x76a264]
  8: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c8ae]
  9: (Thread::_entry_func(void*)+0x23) [0x6f07d5]
  10: (()+0x6d8c) [0x7fd9e6ae7d8c]
  11: (clone()+0x6d) [0x7fd9e552d04d]
• Caught signal (Aborted) *
  in thread 0x7fd9e3229700
  ceph version (commit:)
  1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x79b3ed]
  2: /usr/ceph/bin/cmon() [0x7b4dc3]
  3: (()+0xfc60) [0x7fd9e6af0c60]
  4: (gsignal()+0x35) [0x7fd9e547ad05]
  5: (abort()+0x186) [0x7fd9e547eab6]
  6: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7fd9e5d316dd]
  7: (()+0xb9926) [0x7fd9e5d2f926]
  8: (()+0xb9953) [0x7fd9e5d2f953]
  9: (()+0xb9a5e) [0x7fd9e5d2fa5e]
  10: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x1f3) [0x7b48bb]
  11: (Paxos::handle_collect(MMonPaxos*)+0x697) [0x64ed05]
  12: (Paxos::dispatch(PaxosServiceMessage*)+0x22c) [0x6555b4]
  13: (Monitor::_ms_dispatch(Message*)+0xd23) [0x62fd9d]
  14: (Monitor::ms_dispatch(Message*)+0x3a) [0x63726c]
  15: (Messenger::ms_deliver_dispatch(Message*)+0x70) [0x780622]
  16: (SimpleMessenger::dispatch_entry()+0x810) [0x76a264]
  17: (SimpleMessenger::DispatchThread::entry()+0x2c) [0x61c8ae]
  18: (Thread::_entry_func(void*)+0x23) [0x6f07d5]
  19: (()+0x6d8c) [0x7fd9e6ae7d8c]
  20: (clone()+0x6d) [0x7fd9e552d04d]