Ceph

this is using wip-mon when the cluster is first being setup during pool creation. OSDs were (possibly unrelated) going down as well, but came back up.

From mon.b.log:

     0> 2013-06-04 07:33:57.869882 7f97dd23b700 -1 mon/Paxos.h: In function 'void Paxos::wait_for_readable(Context*)' thread 7f97dd23b700 time 2013-06-04 07:33:57.868692
mon/Paxos.h: 1259: FAILED assert(!is_readable())

 ceph version 0.63-230-ge6d42be (e6d42beddfb18144bdf1715afce676f70d1326fa)
 1: (PGMonitor::check_osd_map(unsigned int)+0x475) [0x54b915]
 2: (OSDMonitor::update_from_paxos(bool*)+0x188e) [0x50daee]
 3: (PaxosService::refresh(bool*)+0x19b) [0x4f535b]
 4: (Monitor::refresh_from_paxos(bool*)+0x57) [0x499077]
 5: (Paxos::finish_proposal()+0x44) [0x4ea734]
 6: (Paxos::handle_accept(MMonPaxos*)+0x775) [0x4eb5e5]
 7: (Paxos::dispatch(PaxosServiceMessage*)+0x24b) [0x4ee27b]
 8: (Monitor::_ms_dispatch(Message*)+0x10c5) [0x4c3be5]
 9: (Monitor::ms_dispatch(Message*)+0x32) [0x4dd712]
 10: (DispatchQueue::entry()+0x3f1) [0x6b60a1]
 11: (DispatchQueue::DispatchThread::entry()+0xd) [0x6436bd]
 12: (()+0x7e9a) [0x7f97e1f9ee9a]
 13: (clone()+0x6d) [0x7f97e054eccd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   0/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 hadoop
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/mon.b.log
--- end dump of recent events ---
2013-06-04 07:33:57.948336 7f97dd23b700 -1 *** Caught signal (Aborted) **
 in thread 7f97dd23b700

 ceph version 0.63-230-ge6d42be (e6d42beddfb18144bdf1715afce676f70d1326fa)
 1: /usr/bin/ceph-mon() [0x59aada]
 2: (()+0xfcb0) [0x7f97e1fa6cb0]
 3: (gsignal()+0x35) [0x7f97e0491425]
 4: (abort()+0x17b) [0x7f97e0494b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f97e0de369d]
 6: (()+0xb5846) [0x7f97e0de1846]
 7: (()+0xb5873) [0x7f97e0de1873]
 8: (()+0xb596e) [0x7f97e0de196e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6540af]
 10: (PGMonitor::check_osd_map(unsigned int)+0x475) [0x54b915]
 11: (OSDMonitor::update_from_paxos(bool*)+0x188e) [0x50daee]
 12: (PaxosService::refresh(bool*)+0x19b) [0x4f535b]
 13: (Monitor::refresh_from_paxos(bool*)+0x57) [0x499077]
 14: (Paxos::finish_proposal()+0x44) [0x4ea734]
 15: (Paxos::handle_accept(MMonPaxos*)+0x775) [0x4eb5e5]
 16: (Paxos::dispatch(PaxosServiceMessage*)+0x24b) [0x4ee27b]
 17: (Monitor::_ms_dispatch(Message*)+0x10c5) [0x4c3be5]
 18: (Monitor::ms_dispatch(Message*)+0x32) [0x4dd712]
 19: (DispatchQueue::entry()+0x3f1) [0x6b60a1]
 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x6436bd]
 21: (()+0x7e9a) [0x7f97e1f9ee9a]
 22: (clone()+0x6d) [0x7f97e054eccd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2013-06-04 07:33:57.948336 7f97dd23b700 -1 *** Caught signal (Aborted) **
 in thread 7f97dd23b700

 ceph version 0.63-230-ge6d42be (e6d42beddfb18144bdf1715afce676f70d1326fa)
 1: /usr/bin/ceph-mon() [0x59aada]
 2: (()+0xfcb0) [0x7f97e1fa6cb0]
 3: (gsignal()+0x35) [0x7f97e0491425]
 4: (abort()+0x17b) [0x7f97e0494b8b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f97e0de369d]
 6: (()+0xb5846) [0x7f97e0de1846]
 7: (()+0xb5873) [0x7f97e0de1873]
 8: (()+0xb596e) [0x7f97e0de196e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6540af]
 10: (PGMonitor::check_osd_map(unsigned int)+0x475) [0x54b915]
 11: (OSDMonitor::update_from_paxos(bool*)+0x188e) [0x50daee]
 12: (PaxosService::refresh(bool*)+0x19b) [0x4f535b]
 13: (Monitor::refresh_from_paxos(bool*)+0x57) [0x499077]
 14: (Paxos::finish_proposal()+0x44) [0x4ea734]
 15: (Paxos::handle_accept(MMonPaxos*)+0x775) [0x4eb5e5]
 16: (Paxos::dispatch(PaxosServiceMessage*)+0x24b) [0x4ee27b]
 17: (Monitor::_ms_dispatch(Message*)+0x10c5) [0x4c3be5]
 18: (Monitor::ms_dispatch(Message*)+0x32) [0x4dd712]
 19: (DispatchQueue::entry()+0x3f1) [0x6b60a1]
 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x6436bd]
 21: (()+0x7e9a) [0x7f97e1f9ee9a]
 22: (clone()+0x6d) [0x7f97e054eccd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.