2012-05-23 19:49:50.265939 7f7c8f34b780 0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-mds, pid 20241 2012-05-23 19:49:50.267753 7f7c8a395700 0 mds.-1.0 ms_handle_connect on 192.168.1.102:6789/0 2012-05-23 19:49:53.268354 7f7c8a395700 0 mds.-1.0 ms_handle_connect on 192.168.1.101:6789/0 2012-05-23 19:49:55.094655 7f7c8a395700 1 mds.-1.0 handle_mds_map standby 2012-05-23 19:50:12.606105 7f7c8a395700 1 mds.0.29 handle_mds_map i am now mds.0.29 2012-05-23 19:50:12.606125 7f7c8a395700 1 mds.0.29 handle_mds_map state change up:standby --> up:replay 2012-05-23 19:50:12.606135 7f7c8a395700 1 mds.0.29 replay_start 2012-05-23 19:50:12.606143 7f7c8a395700 1 mds.0.29 recovery set is 2012-05-23 19:50:12.606145 7f7c8a395700 1 mds.0.29 need osdmap epoch 1741, have 1734 2012-05-23 19:50:12.606148 7f7c8a395700 1 mds.0.29 waiting for osdmap 1741 (which blacklists prior instance) 2012-05-23 19:50:12.606168 7f7c8a395700 1 mds.0.cache handle_mds_failure mds.0 : recovery peers are 2012-05-23 19:50:12.724575 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.102:6801/20407 2012-05-23 19:50:12.724604 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.102:6817/20684 2012-05-23 19:50:12.724984 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.101:6815/17697 2012-05-23 19:50:12.725130 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.101:6810/17611 2012-05-23 19:50:12.725156 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.103:6810/9085 2012-05-23 19:50:12.745135 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.101:6800/17415 2012-05-23 19:50:12.769041 7f7c8a395700 0 mds.0.cache creating system inode with ino:100 2012-05-23 19:50:12.769304 7f7c8a395700 0 mds.0.cache creating system inode with ino:1 2012-05-23 19:50:12.769746 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.101:6805/17500 2012-05-23 19:50:12.782702 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.103:6805/8902 2012-05-23 19:50:12.782728 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.103:6800/8816 2012-05-23 19:50:12.782813 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.103:6817/9170 2012-05-23 19:50:15.263088 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.102:6814/20577 2012-05-23 19:50:20.105956 7f7c87482700 -1 mds/LogEvent.cc: In function 'static LogEvent* LogEvent::decode(ceph::bufferlist&)' thread 7f7c87482700 time 2012-05-23 19:50:20.105525 mds/LogEvent.cc: 95: FAILED assert(p.end()) ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5) 1: (LogEvent::decode(ceph::buffer::list&)+0x29d) [0x6b530d] 2: (MDLog::_replay_thread()+0x668) [0x6a1e68] 3: (MDLog::ReplayThread::entry()+0xd) [0x4d5c1d] 4: (()+0x8ec6) [0x7f7c8e7edec6] 5: (clone()+0x6d) [0x7f7c8d69d51d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -24> 2012-05-23 19:49:50.265939 7f7c8f34b780 0 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5), process ceph-mds, pid 20241 -23> 2012-05-23 19:49:50.267753 7f7c8a395700 0 mds.-1.0 ms_handle_connect on 192.168.1.102:6789/0 -22> 2012-05-23 19:49:53.268354 7f7c8a395700 0 mds.-1.0 ms_handle_connect on 192.168.1.101:6789/0 -21> 2012-05-23 19:49:55.094655 7f7c8a395700 1 mds.-1.0 handle_mds_map standby -20> 2012-05-23 19:50:12.606105 7f7c8a395700 1 mds.0.29 handle_mds_map i am now mds.0.29 -19> 2012-05-23 19:50:12.606125 7f7c8a395700 1 mds.0.29 handle_mds_map state change up:standby --> up:replay -18> 2012-05-23 19:50:12.606135 7f7c8a395700 1 mds.0.29 replay_start -17> 2012-05-23 19:50:12.606143 7f7c8a395700 1 mds.0.29 recovery set is -16> 2012-05-23 19:50:12.606145 7f7c8a395700 1 mds.0.29 need osdmap epoch 1741, have 1734 -15> 2012-05-23 19:50:12.606148 7f7c8a395700 1 mds.0.29 waiting for osdmap 1741 (which blacklists prior instance) -14> 2012-05-23 19:50:12.606168 7f7c8a395700 1 mds.0.cache handle_mds_failure mds.0 : recovery peers are -13> 2012-05-23 19:50:12.724575 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.102:6801/20407 -12> 2012-05-23 19:50:12.724604 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.102:6817/20684 -11> 2012-05-23 19:50:12.724984 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.101:6815/17697 -10> 2012-05-23 19:50:12.725130 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.101:6810/17611 -9> 2012-05-23 19:50:12.725156 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.103:6810/9085 -8> 2012-05-23 19:50:12.745135 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.101:6800/17415 -7> 2012-05-23 19:50:12.769041 7f7c8a395700 0 mds.0.cache creating system inode with ino:100 -6> 2012-05-23 19:50:12.769304 7f7c8a395700 0 mds.0.cache creating system inode with ino:1 -5> 2012-05-23 19:50:12.769746 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.101:6805/17500 -4> 2012-05-23 19:50:12.782702 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.103:6805/8902 -3> 2012-05-23 19:50:12.782728 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.103:6800/8816 -2> 2012-05-23 19:50:12.782813 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.103:6817/9170 -1> 2012-05-23 19:50:15.263088 7f7c8a395700 0 mds.0.29 ms_handle_connect on 192.168.1.102:6814/20577 0> 2012-05-23 19:50:20.105956 7f7c87482700 -1 mds/LogEvent.cc: In function 'static LogEvent* LogEvent::decode(ceph::bufferlist&)' thread 7f7c87482700 time 2012-05-23 19:50:20.105525 mds/LogEvent.cc: 95: FAILED assert(p.end()) ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5) 1: (LogEvent::decode(ceph::buffer::list&)+0x29d) [0x6b530d] 2: (MDLog::_replay_thread()+0x668) [0x6a1e68] 3: (MDLog::ReplayThread::entry()+0xd) [0x4d5c1d] 4: (()+0x8ec6) [0x7f7c8e7edec6] 5: (clone()+0x6d) [0x7f7c8d69d51d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- end dump of recent events --- 2012-05-23 19:50:20.107239 7f7c87482700 -1 *** Caught signal (Aborted) ** in thread 7f7c87482700 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5) 1: /usr/bin/ceph-mds() [0x7e1127] 2: (()+0x10bf0) [0x7f7c8e7f5bf0] 3: (gsignal()+0x35) [0x7f7c8d5eca95] 4: (abort()+0x17b) [0x7f7c8d5edf0b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f7c8df19bed] 6: (()+0xb9da6) [0x7f7c8df17da6] 7: (()+0xb9dd3) [0x7f7c8df17dd3] 8: (()+0xb9ece) [0x7f7c8df17ece] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26f) [0x77827f] 10: (LogEvent::decode(ceph::buffer::list&)+0x29d) [0x6b530d] 11: (MDLog::_replay_thread()+0x668) [0x6a1e68] 12: (MDLog::ReplayThread::entry()+0xd) [0x4d5c1d] 13: (()+0x8ec6) [0x7f7c8e7edec6] 14: (clone()+0x6d) [0x7f7c8d69d51d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2012-05-23 19:50:20.107239 7f7c87482700 -1 *** Caught signal (Aborted) ** in thread 7f7c87482700 ceph version 0.47.1 (commit:f5a9404445e2ed5ec2ee828aa53d73d4a002f7a5) 1: /usr/bin/ceph-mds() [0x7e1127] 2: (()+0x10bf0) [0x7f7c8e7f5bf0] 3: (gsignal()+0x35) [0x7f7c8d5eca95] 4: (abort()+0x17b) [0x7f7c8d5edf0b] 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f7c8df19bed] 6: (()+0xb9da6) [0x7f7c8df17da6] 7: (()+0xb9dd3) [0x7f7c8df17dd3] 8: (()+0xb9ece) [0x7f7c8df17ece] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26f) [0x77827f] 10: (LogEvent::decode(ceph::buffer::list&)+0x29d) [0x6b530d] 11: (MDLog::_replay_thread()+0x668) [0x6a1e68] 12: (MDLog::ReplayThread::entry()+0xd) [0x4d5c1d] 13: (()+0x8ec6) [0x7f7c8e7edec6] 14: (clone()+0x6d) [0x7f7c8d69d51d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- end dump of recent events ---