Ceph

Bug #5246

Updated by Joao Eduardo Luis almost 11 years ago

this is using wip-mon when the cluster is first being setup during pool creation.    OSDs were (possibly unrelated) going down as well, but came back up. 

 From mon.b.log: 

 <pre> 
      

      0> 2013-06-04 07:33:57.869882 7f97dd23b700 -1 mon/Paxos.h: In function 'void Paxos::wait_for_readable(Context*)' thread 7f97dd23b700 time 2013-06-04 07:33:57.868692 
 mon/Paxos.h: 1259: FAILED assert(!is_readable()) 

  ceph version 0.63-230-ge6d42be (e6d42beddfb18144bdf1715afce676f70d1326fa) 
  1: (PGMonitor::check_osd_map(unsigned int)+0x475) [0x54b915] 
  2: (OSDMonitor::update_from_paxos(bool*)+0x188e) [0x50daee] 
  3: (PaxosService::refresh(bool*)+0x19b) [0x4f535b] 
  4: (Monitor::refresh_from_paxos(bool*)+0x57) [0x499077] 
  5: (Paxos::finish_proposal()+0x44) [0x4ea734] 
  6: (Paxos::handle_accept(MMonPaxos*)+0x775) [0x4eb5e5] 
  7: (Paxos::dispatch(PaxosServiceMessage*)+0x24b) [0x4ee27b] 
  8: (Monitor::_ms_dispatch(Message*)+0x10c5) [0x4c3be5] 
  9: (Monitor::ms_dispatch(Message*)+0x32) [0x4dd712] 
  10: (DispatchQueue::entry()+0x3f1) [0x6b60a1] 
  11: (DispatchQueue::DispatchThread::entry()+0xd) [0x6436bd] 
  12: (()+0x7e9a) [0x7f97e1f9ee9a] 
  13: (clone()+0x6d) [0x7f97e054eccd] 
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

 --- logging levels --- 
    0/ 5 none 
    0/ 1 lockdep 
    0/ 1 context 
    1/ 1 crush 
    1/ 5 mds 
    1/ 5 mds_balancer 
    1/ 5 mds_locker 
    1/ 5 mds_log 
    1/ 5 mds_log_expire 
    1/ 5 mds_migrator 
    0/ 1 buffer 
    0/ 1 timer 
    0/ 1 filer 
    0/ 1 striper 
    0/ 1 objecter 
    0/ 5 rados 
    0/ 5 rbd 
    0/ 5 journaler 
    0/ 5 objectcacher 
    0/ 5 client 
    0/ 5 osd 
    0/ 5 optracker 
    0/ 5 objclass 
    1/ 3 filestore 
    1/ 3 journal 
    0/ 5 ms 
    1/ 5 mon 
    0/10 monc 
    0/ 5 paxos 
    0/ 5 tp 
    1/ 5 auth 
    1/ 5 crypto 
    1/ 1 finisher 
    1/ 5 heartbeatmap 
    1/ 5 perfcounter 
    1/ 5 rgw 
    1/ 5 hadoop 
    1/ 5 javaclient 
    1/ 5 asok 
    1/ 1 throttle 
   -2/-2 (syslog threshold) 
   -1/-1 (stderr threshold) 
   max_recent       10000 
   max_new           1000 
   log_file /var/log/ceph/mon.b.log 
 --- end dump of recent events --- 
 2013-06-04 07:33:57.948336 7f97dd23b700 -1 *** Caught signal (Aborted) ** 
  in thread 7f97dd23b700 

  ceph version 0.63-230-ge6d42be (e6d42beddfb18144bdf1715afce676f70d1326fa) 
  1: /usr/bin/ceph-mon() [0x59aada] 
  2: (()+0xfcb0) [0x7f97e1fa6cb0] 
  3: (gsignal()+0x35) [0x7f97e0491425] 
  4: (abort()+0x17b) [0x7f97e0494b8b] 
  5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f97e0de369d] 
  6: (()+0xb5846) [0x7f97e0de1846] 
  7: (()+0xb5873) [0x7f97e0de1873] 
  8: (()+0xb596e) [0x7f97e0de196e] 
  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6540af] 
  10: (PGMonitor::check_osd_map(unsigned int)+0x475) [0x54b915] 
  11: (OSDMonitor::update_from_paxos(bool*)+0x188e) [0x50daee] 
  12: (PaxosService::refresh(bool*)+0x19b) [0x4f535b] 
  13: (Monitor::refresh_from_paxos(bool*)+0x57) [0x499077] 
  14: (Paxos::finish_proposal()+0x44) [0x4ea734] 
  15: (Paxos::handle_accept(MMonPaxos*)+0x775) [0x4eb5e5] 
  16: (Paxos::dispatch(PaxosServiceMessage*)+0x24b) [0x4ee27b] 
  17: (Monitor::_ms_dispatch(Message*)+0x10c5) [0x4c3be5] 
  18: (Monitor::ms_dispatch(Message*)+0x32) [0x4dd712] 
  19: (DispatchQueue::entry()+0x3f1) [0x6b60a1] 
  20: (DispatchQueue::DispatchThread::entry()+0xd) [0x6436bd] 
  21: (()+0x7e9a) [0x7f97e1f9ee9a] 
  22: (clone()+0x6d) [0x7f97e054eccd] 
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 

 --- begin dump of recent events --- 
      0> 2013-06-04 07:33:57.948336 7f97dd23b700 -1 *** Caught signal (Aborted) ** 
  in thread 7f97dd23b700 

  ceph version 0.63-230-ge6d42be (e6d42beddfb18144bdf1715afce676f70d1326fa) 
  1: /usr/bin/ceph-mon() [0x59aada] 
  2: (()+0xfcb0) [0x7f97e1fa6cb0] 
  3: (gsignal()+0x35) [0x7f97e0491425] 
  4: (abort()+0x17b) [0x7f97e0494b8b] 
  5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f97e0de369d] 
  6: (()+0xb5846) [0x7f97e0de1846] 
  7: (()+0xb5873) [0x7f97e0de1873] 
  8: (()+0xb596e) [0x7f97e0de196e] 
  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x6540af] 
  10: (PGMonitor::check_osd_map(unsigned int)+0x475) [0x54b915] 
  11: (OSDMonitor::update_from_paxos(bool*)+0x188e) [0x50daee] 
  12: (PaxosService::refresh(bool*)+0x19b) [0x4f535b] 
  13: (Monitor::refresh_from_paxos(bool*)+0x57) [0x499077] 
  14: (Paxos::finish_proposal()+0x44) [0x4ea734] 
  15: (Paxos::handle_accept(MMonPaxos*)+0x775) [0x4eb5e5] 
  16: (Paxos::dispatch(PaxosServiceMessage*)+0x24b) [0x4ee27b] 
  17: (Monitor::_ms_dispatch(Message*)+0x10c5) [0x4c3be5] 
  18: (Monitor::ms_dispatch(Message*)+0x32) [0x4dd712] 
  19: (DispatchQueue::entry()+0x3f1) [0x6b60a1] 
  20: (DispatchQueue::DispatchThread::entry()+0xd) [0x6436bd] 
  21: (()+0x7e9a) [0x7f97e1f9ee9a] 
  22: (clone()+0x6d) [0x7f97e054eccd] 
  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 
 </pre>

Back