HomeServer2 ~ # ceph-mon -i srv2 -d 2014-01-24 02:49:40.548325 2a1edd54780 0 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mon, pid 4534 starting mon.srv2 rank 1 at 10.0.0.2:6789/0 mon_data /home/ceph_mon/mon.srv2 fsid e6bb627c-0e0f-4176-9e86-9b4bd430b1f2 2014-01-24 02:49:40.614877 2a1edd54780 1 mon.srv2@-1(probing) e1 preinit fsid e6bb627c-0e0f-4176-9e86-9b4bd430b1f2 2014-01-24 02:49:40.616204 2a1edd54780 1 mon.srv2@-1(probing).paxosservice(pgmap 79005..79550) refresh upgraded, format 0 -> 1 2014-01-24 02:49:40.616255 2a1edd54780 1 mon.srv2@-1(probing).pg v0 on_upgrade discarding in-core PGMap 2014-01-24 02:49:40.626296 2a1edd54780 1 mon.srv2@-1(probing).log v74964 unable to write to /var/log/ceph/ceph.log: (2) No such file or directory 2014-01-24 02:49:40.626876 2a1edd54780 1 mon.srv2@-1(probing).paxosservice(auth 251..288) refresh upgraded, format 0 -> 1 mon/AuthMonitor.cc: In function 'virtual void AuthMonitor::update_from_paxos(bool*)' thread 2a1edd54780 time 2014-01-24 02:49:40.627080 mon/AuthMonitor.cc: 153: FAILED assert(ret == 0) ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) 1: (AuthMonitor::update_from_paxos(bool*)+0x2a27) [0x65988bab17] 2: (PaxosService::refresh(bool*)+0x190) [0x6598820930] 3: (Monitor::refresh_from_paxos(bool*)+0x88) [0x65987b95f8] 4: (Monitor::init_paxos()+0x96) [0x65987b9776] 5: (Monitor::preinit()+0x6ee) [0x65987d63fe] 6: (main()+0x1a93) [0x65987a54a3] 7: (__libc_start_main()+0xf5) [0x2a1ebb39965] 8: (()+0x209091) [0x65987a9091] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2014-01-24 02:49:40.629590 2a1edd54780 -1 mon/AuthMonitor.cc: In function 'virtual void AuthMonitor::update_from_paxos(bool*)' thread 2a1edd54780 time 2014-01-24 02:49:40.627080 mon/AuthMonitor.cc: 153: FAILED assert(ret == 0) ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) 1: (AuthMonitor::update_from_paxos(bool*)+0x2a27) [0x65988bab17] 2: (PaxosService::refresh(bool*)+0x190) [0x6598820930] 3: (Monitor::refresh_from_paxos(bool*)+0x88) [0x65987b95f8] 4: (Monitor::init_paxos()+0x96) [0x65987b9776] 5: (Monitor::preinit()+0x6ee) [0x65987d63fe] 6: (main()+0x1a93) [0x65987a54a3] 7: (__libc_start_main()+0xf5) [0x2a1ebb39965] 8: (()+0x209091) [0x65987a9091] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -31> 2014-01-24 02:49:40.539789 2a1edd54780 5 asok(0x659bac3300) register_command perfcounters_dump hook 0x659bac4b10 -30> 2014-01-24 02:49:40.539865 2a1edd54780 5 asok(0x659bac3300) register_command 1 hook 0x659bac4b10 -29> 2014-01-24 02:49:40.539888 2a1edd54780 5 asok(0x659bac3300) register_command perf dump hook 0x659bac4b10 -28> 2014-01-24 02:49:40.539906 2a1edd54780 5 asok(0x659bac3300) register_command perfcounters_schema hook 0x659bac4b10 -27> 2014-01-24 02:49:40.539927 2a1edd54780 5 asok(0x659bac3300) register_command 2 hook 0x659bac4b10 -26> 2014-01-24 02:49:40.539952 2a1edd54780 5 asok(0x659bac3300) register_command perf schema hook 0x659bac4b10 -25> 2014-01-24 02:49:40.539979 2a1edd54780 5 asok(0x659bac3300) register_command config show hook 0x659bac4b10 -24> 2014-01-24 02:49:40.540003 2a1edd54780 5 asok(0x659bac3300) register_command config set hook 0x659bac4b10 -23> 2014-01-24 02:49:40.540032 2a1edd54780 5 asok(0x659bac3300) register_command config get hook 0x659bac4b10 -22> 2014-01-24 02:49:40.540054 2a1edd54780 5 asok(0x659bac3300) register_command log flush hook 0x659bac4b10 -21> 2014-01-24 02:49:40.540076 2a1edd54780 5 asok(0x659bac3300) register_command log dump hook 0x659bac4b10 -20> 2014-01-24 02:49:40.540098 2a1edd54780 5 asok(0x659bac3300) register_command log reopen hook 0x659bac4b10 -19> 2014-01-24 02:49:40.548325 2a1edd54780 0 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mon, pid 4534 -18> 2014-01-24 02:49:40.549032 2a1edd54780 5 asok(0x659bac3300) init /var/run/ceph/ceph-mon.srv2.asok -17> 2014-01-24 02:49:40.549104 2a1edd54780 5 asok(0x659bac3300) bind_and_listen /var/run/ceph/ceph-mon.srv2.asok -16> 2014-01-24 02:49:40.549289 2a1edd54780 5 asok(0x659bac3300) register_command 0 hook 0x659bac7d60 -15> 2014-01-24 02:49:40.549328 2a1edd54780 5 asok(0x659bac3300) register_command version hook 0x659bac7d60 -14> 2014-01-24 02:49:40.549350 2a1edd54780 5 asok(0x659bac3300) register_command git_version hook 0x659bac7d60 -13> 2014-01-24 02:49:40.549375 2a1edd54780 5 asok(0x659bac3300) register_command help hook 0x659bac7d00 -12> 2014-01-24 02:49:40.549399 2a1edd54780 5 asok(0x659bac3300) register_command get_command_descriptions hook 0x659bac7e50 -11> 2014-01-24 02:49:40.549641 2a1ea80c700 5 asok(0x659bac3300) entry start -10> 2014-01-24 02:49:40.614350 2a1edd54780 1 -- 10.0.0.2:6789/0 learned my addr 10.0.0.2:6789/0 -9> 2014-01-24 02:49:40.614424 2a1edd54780 1 accepter.accepter.bind my_inst.addr is 10.0.0.2:6789/0 need_addr=0 -8> 2014-01-24 02:49:40.614507 2a1edd54780 5 adding auth protocol: none -7> 2014-01-24 02:49:40.614540 2a1edd54780 5 adding auth protocol: none -6> 2014-01-24 02:49:40.614877 2a1edd54780 1 mon.srv2@-1(probing) e1 preinit fsid e6bb627c-0e0f-4176-9e86-9b4bd430b1f2 -5> 2014-01-24 02:49:40.616204 2a1edd54780 1 mon.srv2@-1(probing).paxosservice(pgmap 79005..79550) refresh upgraded, format 0 -> 1 -4> 2014-01-24 02:49:40.616255 2a1edd54780 1 mon.srv2@-1(probing).pg v0 on_upgrade discarding in-core PGMap -3> 2014-01-24 02:49:40.624336 2a1edd54780 4 mon.srv2@-1(probing).mds e1 new map -2> 2014-01-24 02:49:40.626296 2a1edd54780 1 mon.srv2@-1(probing).log v74964 unable to write to /var/log/ceph/ceph.log: (2) No such file or directory -1> 2014-01-24 02:49:40.626876 2a1edd54780 1 mon.srv2@-1(probing).paxosservice(auth 251..288) refresh upgraded, format 0 -> 1 0> 2014-01-24 02:49:40.629590 2a1edd54780 -1 mon/AuthMonitor.cc: In function 'virtual void AuthMonitor::update_from_paxos(bool*)' thread 2a1edd54780 time 2014-01-24 02:49:40.627080 mon/AuthMonitor.cc: 153: FAILED assert(ret == 0) ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) 1: (AuthMonitor::update_from_paxos(bool*)+0x2a27) [0x65988bab17] 2: (PaxosService::refresh(bool*)+0x190) [0x6598820930] 3: (Monitor::refresh_from_paxos(bool*)+0x88) [0x65987b95f8] 4: (Monitor::init_paxos()+0x96) [0x65987b9776] 5: (Monitor::preinit()+0x6ee) [0x65987d63fe] 6: (main()+0x1a93) [0x65987a54a3] 7: (__libc_start_main()+0xf5) [0x2a1ebb39965] 8: (()+0x209091) [0x65987a9091] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 10000 max_new 1000 log_file --- end dump of recent events --- terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (Aborted) ** in thread 2a1edd54780 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) 1: (()+0x52c162) [0x6598acc162] 2: (()+0x10aa0) [0x2a1ed936aa0] 3: (gsignal()+0x35) [0x2a1ebb4d2c5] 4: (abort()+0x148) [0x2a1ebb4e748] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x2a1ec43611d] 6: (()+0x65736) [0x2a1ec433736] 7: (()+0x65773) [0x2a1ec433773] 8: (()+0x65ad2) [0x2a1ec433ad2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43f) [0x65989d424f] 10: (AuthMonitor::update_from_paxos(bool*)+0x2a27) [0x65988bab17] 11: (PaxosService::refresh(bool*)+0x190) [0x6598820930] 12: (Monitor::refresh_from_paxos(bool*)+0x88) [0x65987b95f8] 13: (Monitor::init_paxos()+0x96) [0x65987b9776] 14: (Monitor::preinit()+0x6ee) [0x65987d63fe] 15: (main()+0x1a93) [0x65987a54a3] 16: (__libc_start_main()+0xf5) [0x2a1ebb39965] 17: (()+0x209091) [0x65987a9091] 2014-01-24 02:49:40.639752 2a1edd54780 -1 *** Caught signal (Aborted) ** in thread 2a1edd54780 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) 1: (()+0x52c162) [0x6598acc162] 2: (()+0x10aa0) [0x2a1ed936aa0] 3: (gsignal()+0x35) [0x2a1ebb4d2c5] 4: (abort()+0x148) [0x2a1ebb4e748] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x2a1ec43611d] 6: (()+0x65736) [0x2a1ec433736] 7: (()+0x65773) [0x2a1ec433773] 8: (()+0x65ad2) [0x2a1ec433ad2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43f) [0x65989d424f] 10: (AuthMonitor::update_from_paxos(bool*)+0x2a27) [0x65988bab17] 11: (PaxosService::refresh(bool*)+0x190) [0x6598820930] 12: (Monitor::refresh_from_paxos(bool*)+0x88) [0x65987b95f8] 13: (Monitor::init_paxos()+0x96) [0x65987b9776] 14: (Monitor::preinit()+0x6ee) [0x65987d63fe] 15: (main()+0x1a93) [0x65987a54a3] 16: (__libc_start_main()+0xf5) [0x2a1ebb39965] 17: (()+0x209091) [0x65987a9091] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2014-01-24 02:49:40.639752 2a1edd54780 -1 *** Caught signal (Aborted) ** in thread 2a1edd54780 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) 1: (()+0x52c162) [0x6598acc162] 2: (()+0x10aa0) [0x2a1ed936aa0] 3: (gsignal()+0x35) [0x2a1ebb4d2c5] 4: (abort()+0x148) [0x2a1ebb4e748] 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x2a1ec43611d] 6: (()+0x65736) [0x2a1ec433736] 7: (()+0x65773) [0x2a1ec433773] 8: (()+0x65ad2) [0x2a1ec433ad2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43f) [0x65989d424f] 10: (AuthMonitor::update_from_paxos(bool*)+0x2a27) [0x65988bab17] 11: (PaxosService::refresh(bool*)+0x190) [0x6598820930] 12: (Monitor::refresh_from_paxos(bool*)+0x88) [0x65987b95f8] 13: (Monitor::init_paxos()+0x96) [0x65987b9776] 14: (Monitor::preinit()+0x6ee) [0x65987d63fe] 15: (main()+0x1a93) [0x65987a54a3] 16: (__libc_start_main()+0xf5) [0x2a1ebb39965] 17: (()+0x209091) [0x65987a9091] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 10000 max_new 1000 log_file --- end dump of recent events --- Aborted