2014-08-20 06:10:32.763810 7f28a0235780 0 ceph version 0.84-365-g790de97 (790de974a88c1cd26ca5b96d85b6ed8a5a5bfa8b), process ceph-mds, pid 28947 2014-08-20 06:10:33.189355 7f289b2bf700 1 mds.-1.0 handle_mds_map standby 2014-08-20 06:10:33.190643 7f289b2bf700 1 mds.0.1 handle_mds_map i am now mds.0.1 2014-08-20 06:10:33.190649 7f289b2bf700 1 mds.0.1 handle_mds_map state change up:standby --> up:creating 2014-08-20 06:10:33.191077 7f289b2bf700 0 mds.0.cache creating system inode with ino:1 2014-08-20 06:10:33.191427 7f289b2bf700 0 mds.0.cache creating system inode with ino:100 2014-08-20 06:10:33.191437 7f289b2bf700 0 mds.0.cache creating system inode with ino:600 2014-08-20 06:10:33.191583 7f289b2bf700 0 mds.0.cache creating system inode with ino:601 2014-08-20 06:10:33.191658 7f289b2bf700 0 mds.0.cache creating system inode with ino:602 2014-08-20 06:10:33.191732 7f289b2bf700 0 mds.0.cache creating system inode with ino:603 2014-08-20 06:10:33.191801 7f289b2bf700 0 mds.0.cache creating system inode with ino:604 2014-08-20 06:10:33.191860 7f289b2bf700 0 mds.0.cache creating system inode with ino:605 2014-08-20 06:10:33.191924 7f289b2bf700 0 mds.0.cache creating system inode with ino:606 2014-08-20 06:10:33.191989 7f289b2bf700 0 mds.0.cache creating system inode with ino:607 2014-08-20 06:10:33.192054 7f289b2bf700 0 mds.0.cache creating system inode with ino:608 2014-08-20 06:10:33.192107 7f289b2bf700 0 mds.0.cache creating system inode with ino:609 2014-08-20 06:10:33.192178 7f289b2bf700 0 mds.0.cache creating system inode with ino:200 2014-08-20 06:10:33.207783 7f289b2bf700 1 mds.0.1 creating_done 2014-08-20 06:10:34.199567 7f289b2bf700 1 mds.0.1 handle_mds_map i am now mds.0.1 2014-08-20 06:10:34.199576 7f289b2bf700 1 mds.0.1 handle_mds_map state change up:creating --> up:active 2014-08-20 06:10:34.199581 7f289b2bf700 1 mds.0.1 recovery_done -- successful recovery! 2014-08-20 06:10:34.199633 7f289b2bf700 1 mds.0.1 active_start 2014-08-20 06:10:57.797284 7f06fb032780 0 ceph version 0.84-365-g790de97 (790de974a88c1cd26ca5b96d85b6ed8a5a5bfa8b), process ceph-mds, pid 29379 2014-08-20 06:10:58.809791 7f06f60bc700 1 mds.-1.0 handle_mds_map standby 2014-08-20 06:10:58.811041 7f06f60bc700 1 mds.0.2 handle_mds_map i am now mds.0.2 2014-08-20 06:10:58.811045 7f06f60bc700 1 mds.0.2 handle_mds_map state change up:standby --> up:replay 2014-08-20 06:10:58.811047 7f06f60bc700 1 mds.0.2 replay_start 2014-08-20 06:10:58.811054 7f06f60bc700 1 mds.0.2 recovery set is 2014-08-20 06:10:58.811056 7f06f60bc700 1 mds.0.2 need osdmap epoch 8, have 7 2014-08-20 06:10:58.811058 7f06f60bc700 1 mds.0.2 waiting for osdmap 8 (which blacklists prior instance) 2014-08-20 06:10:58.823540 7f06f25b2700 0 mds.0.cache creating system inode with ino:100 2014-08-20 06:10:58.823852 7f06f25b2700 0 mds.0.cache creating system inode with ino:1 2014-08-20 06:10:58.826634 7f06f25b2700 1 mds.0.2 replay_done 2014-08-20 06:10:58.826646 7f06f25b2700 1 mds.0.2 making mds journal writeable 2014-08-20 06:10:59.815685 7f06f60bc700 1 mds.0.2 handle_mds_map i am now mds.0.2 2014-08-20 06:10:59.815708 7f06f60bc700 1 mds.0.2 handle_mds_map state change up:replay --> up:reconnect 2014-08-20 06:10:59.815713 7f06f60bc700 1 mds.0.2 reconnect_start 2014-08-20 06:10:59.815717 7f06f60bc700 1 mds.0.2 reopen_log 2014-08-20 06:10:59.815730 7f06f60bc700 1 mds.0.2 reconnect_done 2014-08-20 06:11:00.816610 7f06f60bc700 1 mds.0.2 handle_mds_map i am now mds.0.2 2014-08-20 06:11:00.816613 7f06f60bc700 1 mds.0.2 handle_mds_map state change up:reconnect --> up:rejoin 2014-08-20 06:11:00.816616 7f06f60bc700 1 mds.0.2 rejoin_start 2014-08-20 06:11:00.816629 7f06f60bc700 1 mds.0.2 rejoin_joint_start 2014-08-20 06:11:00.816691 7f06f60bc700 1 mds.0.2 rejoin_done 2014-08-20 06:11:01.819094 7f06f60bc700 1 mds.0.2 handle_mds_map i am now mds.0.2 2014-08-20 06:11:01.819101 7f06f60bc700 1 mds.0.2 handle_mds_map state change up:rejoin --> up:active 2014-08-20 06:11:01.819106 7f06f60bc700 1 mds.0.2 recovery_done -- successful recovery! 2014-08-20 06:11:01.819295 7f06f60bc700 1 mds.0.2 active_start 2014-08-20 06:11:01.819345 7f06f60bc700 1 mds.0.2 cluster recovered. 2014-08-20 06:11:18.630494 7f06f70be700 1 mds.0.2 asok_command: session ls 2014-08-20 06:11:32.679569 7f8f1989b780 0 ceph version 0.84-365-g790de97 (790de974a88c1cd26ca5b96d85b6ed8a5a5bfa8b), process ceph-mds, pid 29438 2014-08-20 06:11:33.694861 7f8f14925700 1 mds.-1.0 handle_mds_map standby 2014-08-20 06:11:33.696151 7f8f14925700 1 mds.0.3 handle_mds_map i am now mds.0.3 2014-08-20 06:11:33.696158 7f8f14925700 1 mds.0.3 handle_mds_map state change up:standby --> up:replay 2014-08-20 06:11:33.696161 7f8f14925700 1 mds.0.3 replay_start 2014-08-20 06:11:33.696169 7f8f14925700 1 mds.0.3 recovery set is 2014-08-20 06:11:33.696172 7f8f14925700 1 mds.0.3 need osdmap epoch 9, have 8 2014-08-20 06:11:33.696174 7f8f14925700 1 mds.0.3 waiting for osdmap 9 (which blacklists prior instance) 2014-08-20 06:11:33.712745 7f8f10e1b700 0 mds.0.cache creating system inode with ino:100 2014-08-20 06:11:33.713081 7f8f10e1b700 0 mds.0.cache creating system inode with ino:1 2014-08-20 06:11:33.716840 7f8f10e1b700 1 mds.0.3 replay_done 2014-08-20 06:11:33.716854 7f8f10e1b700 1 mds.0.3 making mds journal writeable 2014-08-20 06:11:34.703320 7f8f14925700 1 mds.0.3 handle_mds_map i am now mds.0.3 2014-08-20 06:11:34.703326 7f8f14925700 1 mds.0.3 handle_mds_map state change up:replay --> up:reconnect 2014-08-20 06:11:34.703329 7f8f14925700 1 mds.0.3 reconnect_start 2014-08-20 06:11:34.703331 7f8f14925700 1 mds.0.3 reopen_log 2014-08-20 06:11:34.703340 7f8f14925700 1 mds.0.3 reconnect_done 2014-08-20 06:11:34.706040 7f8f14925700 1 mds.0.server session is closed, ignoring reconnect, sending close 2014-08-20 06:11:34.706076 7f8f14925700 0 log [INF] : denied reconnect attempt (mds is up:reconnect) from client.4122 10.214.137.130:0/29747 (session is closed) 2014-08-20 06:11:34.707038 7f8f0fa15700 0 -- 10.214.136.128:6805/29438 >> 10.214.137.130:0/29747 pipe(0x2848340 sd=20 :6805 s=2 pgs=4 cs=1 l=0 c=0x283a420).fault, server, going to standby 2014-08-20 06:11:35.706406 7f8f14925700 1 mds.0.3 handle_mds_map i am now mds.0.3 2014-08-20 06:11:35.706412 7f8f14925700 1 mds.0.3 handle_mds_map state change up:reconnect --> up:rejoin 2014-08-20 06:11:35.706415 7f8f14925700 1 mds.0.3 rejoin_start 2014-08-20 06:11:35.706430 7f8f14925700 1 mds.0.3 rejoin_joint_start 2014-08-20 06:11:35.706481 7f8f14925700 1 mds.0.3 rejoin_done 2014-08-20 06:11:36.707298 7f8f14925700 1 mds.0.3 handle_mds_map i am now mds.0.3 2014-08-20 06:11:36.707305 7f8f14925700 1 mds.0.3 handle_mds_map state change up:rejoin --> up:active 2014-08-20 06:11:36.707308 7f8f14925700 1 mds.0.3 recovery_done -- successful recovery! 2014-08-20 06:11:36.707455 7f8f14925700 1 mds.0.3 active_start 2014-08-20 06:11:36.707470 7f8f14925700 1 mds.0.3 cluster recovered. 2014-08-20 06:11:36.709031 7f8f0f813700 0 -- 10.214.136.128:6805/29438 >> 10.214.137.130:0/29747 pipe(0x287db80 sd=21 :6805 s=0 pgs=0 cs=0 l=0 c=0x283adc0).accept connect_seq 0 vs existing 1 state standby 2014-08-20 06:11:36.709058 7f8f0f813700 0 -- 10.214.136.128:6805/29438 >> 10.214.137.130:0/29747 pipe(0x287db80 sd=21 :6805 s=0 pgs=0 cs=0 l=0 c=0x283adc0).accept peer reset, then tried to connect to us, replacing 2014-08-20 06:11:44.704737 7f8f0f813700 0 -- 10.214.136.128:6805/29438 >> 10.214.137.130:0/29747 pipe(0x28ba580 sd=20 :6805 s=2 pgs=6 cs=1 l=0 c=0x283ab00).fault with nothing to send, going to standby 2014-08-20 06:11:51.351509 7f8f15927700 1 mds.0.3 asok_command: session evict 2014-08-20 06:12:00.679135 7f8f0f712700 0 -- 10.214.136.128:6805/29438 >> 10.214.137.130:0/29770 pipe(0x28ba840 sd=22 :6805 s=2 pgs=2 cs=1 l=0 c=0x283ac60).fault with nothing to send, going to standby 2014-08-20 06:12:08.475069 7f3301088780 0 ceph version 0.84-365-g790de97 (790de974a88c1cd26ca5b96d85b6ed8a5a5bfa8b), process ceph-mds, pid 29505 2014-08-20 06:12:09.491229 7f32fc112700 1 mds.-1.0 handle_mds_map standby 2014-08-20 06:12:09.492461 7f32fc112700 1 mds.0.4 handle_mds_map i am now mds.0.4 2014-08-20 06:12:09.492466 7f32fc112700 1 mds.0.4 handle_mds_map state change up:standby --> up:replay 2014-08-20 06:12:09.492469 7f32fc112700 1 mds.0.4 replay_start 2014-08-20 06:12:09.492477 7f32fc112700 1 mds.0.4 recovery set is 2014-08-20 06:12:09.492479 7f32fc112700 1 mds.0.4 need osdmap epoch 10, have 9 2014-08-20 06:12:09.492482 7f32fc112700 1 mds.0.4 waiting for osdmap 10 (which blacklists prior instance) 2014-08-20 06:12:09.503491 7f32f8608700 0 mds.0.cache creating system inode with ino:100 2014-08-20 06:12:09.503669 7f32f8608700 0 mds.0.cache creating system inode with ino:1 2014-08-20 06:12:09.506972 7f32f8608700 1 mds.0.4 replay_done 2014-08-20 06:12:09.506986 7f32f8608700 1 mds.0.4 making mds journal writeable 2014-08-20 06:12:10.495302 7f32fc112700 1 mds.0.4 handle_mds_map i am now mds.0.4 2014-08-20 06:12:10.495309 7f32fc112700 1 mds.0.4 handle_mds_map state change up:replay --> up:reconnect 2014-08-20 06:12:10.495312 7f32fc112700 1 mds.0.4 reconnect_start 2014-08-20 06:12:10.495314 7f32fc112700 1 mds.0.4 reopen_log 2014-08-20 06:12:10.495322 7f32fc112700 1 mds.0.server reconnect_clients -- 1 sessions 2014-08-20 06:12:10.497631 7f32fc112700 1 mds.0.server session is closed, ignoring reconnect, sending close 2014-08-20 06:12:10.497667 7f32fc112700 0 log [INF] : denied reconnect attempt (mds is up:reconnect) from client.4126 10.214.137.130:0/29908 (session is closed) 2014-08-20 06:12:10.498447 7f32f7202700 0 -- 10.214.136.128:6805/29505 >> 10.214.137.130:0/29908 pipe(0x2d80340 sd=20 :6805 s=2 pgs=4 cs=1 l=0 c=0x2d72420).fault, server, going to standby 2014-08-20 06:12:58.485365 7f32f980c700 1 mds.0.server reconnect gave up on client.4123 10.214.137.130:0/29770 2014-08-20 06:12:58.485403 7f32f980c700 1 mds.0.4 reconnect_done 2014-08-20 06:12:58.495556 7f32fc112700 -1 *** Caught signal (Aborted) ** in thread 7f32fc112700 ceph version 0.84-365-g790de97 (790de974a88c1cd26ca5b96d85b6ed8a5a5bfa8b) 1: ceph-mds() [0x810cfa] 2: (()+0xfcb0) [0x7f3300a01cb0] 3: (gsignal()+0x35) [0x7f32ff4f0425] 4: (abort()+0x17b) [0x7f32ff4f3b8b] 5: (()+0x2f0ee) [0x7f32ff4e90ee] 6: (()+0x2f192) [0x7f32ff4e9192] 7: ceph-mds() [0x57215d] 8: (Server::_session_logged(Session*, unsigned long, bool, unsigned long, interval_set&, unsigned long)+0x151b) [0x5adceb] 9: (C_MDS_session_finish::finish(int)+0x38) [0x5ff478] 10: (Context::complete(int)+0x9) [0x595859] 11: (finish_contexts(CephContext*, std::list >&, int)+0x95) [0x597065] 12: (Journaler::_finish_flush(int, unsigned long, utime_t)+0x1b2) [0x7a3b82] 13: (Context::complete(int)+0x9) [0x595859] 14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x1073) [0x7be213] 15: (MDS::handle_core_message(Message*)+0x5b8) [0x592948] 16: (MDS::_dispatch(Message*)+0x2f) [0x59308f] 17: (MDS::ms_dispatch(Message*)+0x1e4) [0x594af4] 18: (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x9c3f87] 19: (DispatchQueue::entry()+0x4b2) [0x9c0d22] 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x8d5bcd] 21: (()+0x7e9a) [0x7f33009f9e9a] 22: (clone()+0x6d) [0x7f32ff5ae3fd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -260> 2014-08-20 06:12:08.471138 7f3301088780 5 asok(0x2d59000) register_command perfcounters_dump hook 0x2d50010 -259> 2014-08-20 06:12:08.471183 7f3301088780 5 asok(0x2d59000) register_command 1 hook 0x2d50010 -258> 2014-08-20 06:12:08.471190 7f3301088780 5 asok(0x2d59000) register_command perf dump hook 0x2d50010 -257> 2014-08-20 06:12:08.471210 7f3301088780 5 asok(0x2d59000) register_command perfcounters_schema hook 0x2d50010 -256> 2014-08-20 06:12:08.471220 7f3301088780 5 asok(0x2d59000) register_command 2 hook 0x2d50010 -255> 2014-08-20 06:12:08.471224 7f3301088780 5 asok(0x2d59000) register_command perf schema hook 0x2d50010 -254> 2014-08-20 06:12:08.471232 7f3301088780 5 asok(0x2d59000) register_command config show hook 0x2d50010 -253> 2014-08-20 06:12:08.471242 7f3301088780 5 asok(0x2d59000) register_command config set hook 0x2d50010 -252> 2014-08-20 06:12:08.471258 7f3301088780 5 asok(0x2d59000) register_command config get hook 0x2d50010 -251> 2014-08-20 06:12:08.471266 7f3301088780 5 asok(0x2d59000) register_command log flush hook 0x2d50010 -250> 2014-08-20 06:12:08.471271 7f3301088780 5 asok(0x2d59000) register_command log dump hook 0x2d50010 -249> 2014-08-20 06:12:08.471278 7f3301088780 5 asok(0x2d59000) register_command log reopen hook 0x2d50010 -248> 2014-08-20 06:12:08.475069 7f3301088780 0 ceph version 0.84-365-g790de97 (790de974a88c1cd26ca5b96d85b6ed8a5a5bfa8b), process ceph-mds, pid 29505 -247> 2014-08-20 06:12:08.475413 7f3301088780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6805/29505 need_addr=1 -246> 2014-08-20 06:12:08.477570 7f3301088780 5 asok(0x2d59000) init /var/run/ceph/ceph-mds.a.asok -245> 2014-08-20 06:12:08.477590 7f3301088780 5 asok(0x2d59000) bind_and_listen /var/run/ceph/ceph-mds.a.asok -244> 2014-08-20 06:12:08.477727 7f3301088780 5 asok(0x2d59000) register_command 0 hook 0x2d4f0b8 -243> 2014-08-20 06:12:08.477737 7f3301088780 5 asok(0x2d59000) register_command version hook 0x2d4f0b8 -242> 2014-08-20 06:12:08.477740 7f3301088780 5 asok(0x2d59000) register_command git_version hook 0x2d4f0b8 -241> 2014-08-20 06:12:08.477742 7f3301088780 5 asok(0x2d59000) register_command help hook 0x2d500c0 -240> 2014-08-20 06:12:08.477744 7f3301088780 5 asok(0x2d59000) register_command get_command_descriptions hook 0x2d500b0 -239> 2014-08-20 06:12:08.477770 7f3301088780 10 monclient(hunting): build_initial_monmap -238> 2014-08-20 06:12:08.477798 7f3301088780 1 -- 0.0.0.0:6805/29505 messenger.start -237> 2014-08-20 06:12:08.477826 7f3301088780 5 adding auth protocol: cephx -236> 2014-08-20 06:12:08.477829 7f3301088780 5 adding auth protocol: cephx -235> 2014-08-20 06:12:08.477849 7f32fd114700 5 asok(0x2d59000) entry start -234> 2014-08-20 06:12:08.477888 7f3301088780 1 accepter.accepter.start -233> 2014-08-20 06:12:08.477917 7f3301088780 10 monclient(hunting): init -232> 2014-08-20 06:12:08.477921 7f3301088780 5 adding auth protocol: cephx -231> 2014-08-20 06:12:08.477922 7f3301088780 10 monclient(hunting): auth_supported 2 method cephx -230> 2014-08-20 06:12:08.478039 7f3301088780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-a/keyring -229> 2014-08-20 06:12:08.478087 7f3301088780 10 monclient(hunting): _reopen_session rank -1 name -228> 2014-08-20 06:12:08.478119 7f3301088780 10 monclient(hunting): picked mon.a con 0x2d722c0 addr 10.214.136.128:6789/0 -227> 2014-08-20 06:12:08.478134 7f3301088780 10 monclient(hunting): _send_mon_message to mon.a at 10.214.136.128:6789/0 -226> 2014-08-20 06:12:08.478137 7f3301088780 1 -- 0.0.0.0:6805/29505 --> 10.214.136.128:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x2d58000 con 0x2d722c0 -225> 2014-08-20 06:12:08.478168 7f3301088780 10 monclient(hunting): renew_subs -224> 2014-08-20 06:12:08.478530 7f3301080700 1 -- 10.214.136.128:6805/29505 learned my addr 10.214.136.128:6805/29505 -223> 2014-08-20 06:12:08.478948 7f32fc112700 5 mds.-1.0 ms_handle_connect on 10.214.136.128:6789/0 -222> 2014-08-20 06:12:08.479797 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 1 ==== mon_map magic: 0 v1 ==== 191+0+0 (675201266 0 0) 0x2d92a00 con 0x2d722c0 -221> 2014-08-20 06:12:08.479849 7f32fc112700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -220> 2014-08-20 06:12:08.479862 7f32fc112700 10 monclient(hunting): got monmap 1, mon.a is now rank 0 -219> 2014-08-20 06:12:08.479865 7f32fc112700 10 monclient(hunting): dump: epoch 1 fsid fa9c9cd9-fc25-4986-9c97-b250911a7a53 last_changed 2014-08-20 06:09:43.291876 created 2014-08-20 06:09:43.291876 0: 10.214.136.128:6789/0 mon.a -218> 2014-08-20 06:12:08.479909 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2444827731 0 0) 0x2d58480 con 0x2d722c0 -217> 2014-08-20 06:12:08.479947 7f32fc112700 10 monclient(hunting): my global_id is 4125 -216> 2014-08-20 06:12:08.480065 7f32fc112700 10 monclient(hunting): _send_mon_message to mon.a at 10.214.136.128:6789/0 -215> 2014-08-20 06:12:08.480073 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x2d586c0 con 0x2d722c0 -214> 2014-08-20 06:12:08.480840 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 206+0+0 (1919330145 0 0) 0x2d586c0 con 0x2d722c0 -213> 2014-08-20 06:12:08.481001 7f32fc112700 10 monclient(hunting): _send_mon_message to mon.a at 10.214.136.128:6789/0 -212> 2014-08-20 06:12:08.481012 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2d58480 con 0x2d722c0 -211> 2014-08-20 06:12:08.482080 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (2753178087 0 0) 0x2d58000 con 0x2d722c0 -210> 2014-08-20 06:12:08.482237 7f32fc112700 1 monclient(hunting): found mon.a -209> 2014-08-20 06:12:08.482246 7f32fc112700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -208> 2014-08-20 06:12:08.482254 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x2d92800 con 0x2d722c0 -207> 2014-08-20 06:12:08.482283 7f32fc112700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2014-08-20 06:11:38.482282) -206> 2014-08-20 06:12:08.482295 7f32fc112700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -205> 2014-08-20 06:12:08.482305 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2d586c0 con 0x2d722c0 -204> 2014-08-20 06:12:08.482399 7f3301088780 5 monclient: authenticate success, global_id 4125 -203> 2014-08-20 06:12:08.482419 7f3301088780 10 monclient: wait_auth_rotating waiting (until 2014-08-20 06:12:38.482418) -202> 2014-08-20 06:12:08.482783 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 5 ==== mon_map magic: 0 v1 ==== 191+0+0 (675201266 0 0) 0x2d92e00 con 0x2d722c0 -201> 2014-08-20 06:12:08.482816 7f32fc112700 10 monclient: handle_monmap mon_map magic: 0 v1 -200> 2014-08-20 06:12:08.482825 7f32fc112700 10 monclient: got monmap 1, mon.a is now rank 0 -199> 2014-08-20 06:12:08.482839 7f32fc112700 10 monclient: dump: epoch 1 fsid fa9c9cd9-fc25-4986-9c97-b250911a7a53 last_changed 2014-08-20 06:09:43.291876 created 2014-08-20 06:09:43.291876 0: 10.214.136.128:6789/0 mon.a -198> 2014-08-20 06:12:08.482867 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3368627867 0 0) 0x2d71a80 con 0x2d722c0 -197> 2014-08-20 06:12:08.482883 7f32fc112700 10 monclient: handle_subscribe_ack sent 2014-08-20 06:12:08.478170 renew after 2014-08-20 06:14:38.478170 -196> 2014-08-20 06:12:08.482901 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (3263831720 0 0) 0x2d58d80 con 0x2d722c0 -195> 2014-08-20 06:12:08.482980 7f32fc112700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-08-20 06:11:38.482980) -194> 2014-08-20 06:12:08.483072 7f3301088780 10 monclient: wait_auth_rotating done -193> 2014-08-20 06:12:08.483117 7f3301088780 5 asok(0x2d59000) register_command objecter_requests hook 0x2d50110 -192> 2014-08-20 06:12:08.483137 7f3301088780 10 monclient: renew_subs -191> 2014-08-20 06:12:08.483145 7f3301088780 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -190> 2014-08-20 06:12:08.483154 7f3301088780 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x2d92600 con 0x2d722c0 -189> 2014-08-20 06:12:08.483174 7f3301088780 10 monclient: renew_subs -188> 2014-08-20 06:12:08.483179 7f3301088780 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -187> 2014-08-20 06:12:08.483186 7f3301088780 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x2d92400 con 0x2d722c0 -186> 2014-08-20 06:12:08.483768 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 8 ==== osd_map(9..9 src has 1..9) v3 ==== 3652+0+0 (3200850769 0 0) 0x2d58d80 con 0x2d722c0 -185> 2014-08-20 06:12:08.483981 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3368627867 0 0) 0x2d71e00 con 0x2d722c0 -184> 2014-08-20 06:12:08.484008 7f32fc112700 10 monclient: handle_subscribe_ack sent 2014-08-20 06:12:08.483143 renew after 2014-08-20 06:14:38.483143 -183> 2014-08-20 06:12:08.484022 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 10 ==== mdsmap(e 16) v1 ==== 661+0+0 (4218713730 0 0) 0x2d58000 con 0x2d722c0 -182> 2014-08-20 06:12:08.484037 7f32fc112700 5 mds.-1.0 handle_mds_map epoch 16 from mon.0 -181> 2014-08-20 06:12:08.484097 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 11 ==== osd_map(9..9 src has 1..9) v3 ==== 3652+0+0 (3200850769 0 0) 0x2d58b40 con 0x2d722c0 -180> 2014-08-20 06:12:08.484122 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3368627867 0 0) 0x2d71c40 con 0x2d722c0 -179> 2014-08-20 06:12:08.484133 7f32fc112700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring -178> 2014-08-20 06:12:08.484232 7f3301088780 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -177> 2014-08-20 06:12:08.484252 7f3301088780 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:boot seq 1 v16) v2 -- ?+0 0x2d7fb00 con 0x2d722c0 -176> 2014-08-20 06:12:08.484318 7f3301088780 5 asok(0x2d59000) register_command status hook 0x2d50290 -175> 2014-08-20 06:12:08.484331 7f3301088780 5 asok(0x2d59000) register_command dump_ops_in_flight hook 0x2d50290 -174> 2014-08-20 06:12:08.484337 7f3301088780 5 asok(0x2d59000) register_command dump_historic_ops hook 0x2d50290 -173> 2014-08-20 06:12:08.484345 7f3301088780 5 asok(0x2d59000) register_command session evict hook 0x2d50290 -172> 2014-08-20 06:12:08.484352 7f3301088780 5 asok(0x2d59000) register_command session ls hook 0x2d50290 -171> 2014-08-20 06:12:09.465712 7f32f880a700 1 -- 10.214.136.128:6805/29505 >> :/0 pipe(0x2d80340 sd=17 :6805 s=0 pgs=0 cs=0 l=0 c=0x2d72420).accept sd=17 10.214.137.130:43702/0 -170> 2014-08-20 06:12:09.466208 7f32f880a700 2 -- 10.214.136.128:6805/29505 >> :/0 pipe(0x2d80340 sd=17 :6805 s=4 pgs=0 cs=0 l=0 c=0x2d72420).fault (0) Success -169> 2014-08-20 06:12:09.466321 7f32fc112700 5 mds.-1.-1 ms_handle_reset on :/0 -168> 2014-08-20 06:12:09.466555 7f32f880a700 1 -- 10.214.136.128:6805/29505 >> :/0 pipe(0x2d80600 sd=17 :6805 s=0 pgs=0 cs=0 l=0 c=0x2d72580).accept sd=17 10.214.137.130:43703/0 -167> 2014-08-20 06:12:09.466950 7f32f880a700 2 -- 10.214.136.128:6805/29505 >> :/0 pipe(0x2d80600 sd=17 :6805 s=4 pgs=0 cs=0 l=0 c=0x2d72580).fault (0) Success -166> 2014-08-20 06:12:09.467043 7f32fc112700 5 mds.-1.-1 ms_handle_reset on :/0 -165> 2014-08-20 06:12:09.491135 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 13 ==== mdsmap(e 17) v1 ==== 653+0+0 (1166991204 0 0) 0x2d58900 con 0x2d722c0 -164> 2014-08-20 06:12:09.491170 7f32fc112700 5 mds.-1.-1 handle_mds_map epoch 17 from mon.0 -163> 2014-08-20 06:12:09.491210 7f32fc112700 1 -- 10.214.136.128:6805/29505 mark_down 10.214.136.128:6805/29438 -- pipe dne -162> 2014-08-20 06:12:09.491229 7f32fc112700 1 mds.-1.0 handle_mds_map standby -161> 2014-08-20 06:12:09.492397 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 14 ==== mdsmap(e 18) v1 ==== 661+0+0 (2477667411 0 0) 0x2da5900 con 0x2d722c0 -160> 2014-08-20 06:12:09.492430 7f32fc112700 5 mds.-1.0 handle_mds_map epoch 18 from mon.0 -159> 2014-08-20 06:12:09.492461 7f32fc112700 1 mds.0.4 handle_mds_map i am now mds.0.4 -158> 2014-08-20 06:12:09.492466 7f32fc112700 1 mds.0.4 handle_mds_map state change up:standby --> up:replay -157> 2014-08-20 06:12:09.492469 7f32fc112700 1 mds.0.4 replay_start -156> 2014-08-20 06:12:09.492477 7f32fc112700 1 mds.0.4 recovery set is -155> 2014-08-20 06:12:09.492479 7f32fc112700 1 mds.0.4 need osdmap epoch 10, have 9 -154> 2014-08-20 06:12:09.492482 7f32fc112700 1 mds.0.4 waiting for osdmap 10 (which blacklists prior instance) -153> 2014-08-20 06:12:09.492489 7f32fc112700 10 monclient: renew_subs -152> 2014-08-20 06:12:09.492493 7f32fc112700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -151> 2014-08-20 06:12:09.492499 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mon_subscribe({mdsmap=19+,monmap=2+,osdmap=10}) v2 -- ?+0 0x2d92600 con 0x2d722c0 -150> 2014-08-20 06:12:09.493231 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 15 ==== osd_map(10..10 src has 1..10) v3 ==== 350+0+0 (667294219 0 0) 0x2da56c0 con 0x2d722c0 -149> 2014-08-20 06:12:09.493301 7f32fc112700 2 mds.0.4 boot_start 0: opening inotable -148> 2014-08-20 06:12:09.493404 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6800/28669 -- osd_op(mds.0.4:1 mds0_inotable [read 0~0] 1.b852b893 ack+read+known_if_redirected e10) v4 -- ?+0 0x2d8b000 con 0x2d726e0 -147> 2014-08-20 06:12:09.493437 7f32fc112700 2 mds.0.4 boot_start 0: opening sessionmap -146> 2014-08-20 06:12:09.493530 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:2 mds0_sessionmap [read 0~0] 1.3270c60b ack+read+known_if_redirected e10) v4 -- ?+0 0x2d8bc80 con 0x2d729a0 -145> 2014-08-20 06:12:09.493558 7f32fc112700 2 mds.0.4 boot_start 0: opening mds log -144> 2014-08-20 06:12:09.493561 7f32fc112700 5 mds.0.log open discovering log bounds -143> 2014-08-20 06:12:09.493623 7f32fc112700 2 mds.0.4 boot_start 0: opening snap table -142> 2014-08-20 06:12:09.493690 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6805/29464 -- osd_op(mds.0.4:3 mds_snaptable [read 0~0] 1.d90270ad ack+read+known_if_redirected e10) v4 -- ?+0 0x2da9500 con 0x2d72840 -141> 2014-08-20 06:12:09.493715 7f32f8608700 4 mds.0.journalpointer Reading journal pointer '400.00000000' -140> 2014-08-20 06:12:09.493730 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 16 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3368627867 0 0) 0x2d9d8c0 con 0x2d722c0 -139> 2014-08-20 06:12:09.493747 7f32fc112700 10 monclient: handle_subscribe_ack sent 2014-08-20 06:12:09.492491 renew after 2014-08-20 06:14:39.492491 -138> 2014-08-20 06:12:09.493766 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:4 400.00000000 [read 0~0] 1.64e96f8f ack+read+known_if_redirected e10) v4 -- ?+0 0x2da9780 con 0x2d729a0 -137> 2014-08-20 06:12:09.494302 7f32fc112700 5 mds.0.4 ms_handle_connect on 10.214.136.128:6800/28669 -136> 2014-08-20 06:12:09.495135 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.0 10.214.136.128:6800/28669 1 ==== osd_op_reply(1 mds0_inotable [read 0~34] v0'0 uv1 ondisk = 0) v6 ==== 180+0+34 (1595012767 0 1054821296) 0x2d80b80 con 0x2d726e0 -135> 2014-08-20 06:12:09.495211 7f32fc112700 5 mds.0.4 ms_handle_connect on 10.214.137.130:6805/29464 -134> 2014-08-20 06:12:09.495243 7f32fc112700 5 mds.0.4 ms_handle_connect on 10.214.137.130:6800/29383 -133> 2014-08-20 06:12:09.500518 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 1 ==== osd_op_reply(4 400.00000000 [read 0~22] v0'0 uv1 ondisk = 0) v6 ==== 179+0+22 (1573933699 0 3825830296) 0x2db4000 con 0x2d729a0 -132> 2014-08-20 06:12:09.500597 7f32f8608700 1 mds.0.journaler(ro) recover start -131> 2014-08-20 06:12:09.500606 7f32f8608700 1 mds.0.journaler(ro) read_head -130> 2014-08-20 06:12:09.500637 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:5 200.00000000 [read 0~0] 1.844f3494 ack+read+known_if_redirected e10) v4 -- ?+0 0x2da9780 con 0x2d729a0 -129> 2014-08-20 06:12:09.500667 7f32f8608700 4 mds.0.log Waiting for journal 200 to recover... -128> 2014-08-20 06:12:09.501460 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.2 10.214.137.130:6805/29464 1 ==== osd_op_reply(3 mds_snaptable [read 0~46] v0'0 uv1 ondisk = 0) v6 ==== 180+0+46 (408924715 0 2603725198) 0x2db42c0 con 0x2d72840 -127> 2014-08-20 06:12:09.502039 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 2 ==== osd_op_reply(5 200.00000000 [read 0~90] v0'0 uv5 ondisk = 0) v6 ==== 179+0+90 (3927939839 0 413656017) 0x2db4840 con 0x2d729a0 -126> 2014-08-20 06:12:09.502094 7f32fc112700 1 mds.0.journaler(ro) _finish_read_head loghead(trim 4194304, expire 4194304, write 4211946, stream_format 1). probing for end of log (from 4211946)... -125> 2014-08-20 06:12:09.502103 7f32fc112700 1 mds.0.journaler(ro) probing for end of the log -124> 2014-08-20 06:12:09.502151 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:6 200.00000001 [stat] 1.6e5f474 ack+read+rwordered+known_if_redirected e10) v4 -- ?+0 0x2da9280 con 0x2d729a0 -123> 2014-08-20 06:12:09.502190 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:7 200.00000002 [stat] 1.eb272dbb ack+read+rwordered+known_if_redirected e10) v4 -- ?+0 0x2da9000 con 0x2d729a0 -122> 2014-08-20 06:12:09.503177 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 3 ==== osd_op_reply(6 200.00000001 [stat] v0'0 uv24 ondisk = 0) v6 ==== 179+0+16 (4210735968 0 2939838504) 0x2db4580 con 0x2d729a0 -121> 2014-08-20 06:12:09.503331 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 4 ==== osd_op_reply(2 mds0_sessionmap [read 0~22] v0'0 uv1 ondisk = 0) v6 ==== 182+0+22 (784851035 0 3628653333) 0x2db42c0 con 0x2d729a0 -120> 2014-08-20 06:12:09.503395 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 5 ==== osd_op_reply(7 200.00000002 [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v6 ==== 179+0+0 (795473583 0 0) 0x2db4000 con 0x2d729a0 -119> 2014-08-20 06:12:09.503430 7f32fc112700 1 mds.0.journaler(ro) _finish_probe_end write_pos = 4214472 (header had 4211946). recovered. -118> 2014-08-20 06:12:09.503457 7f32f8608700 4 mds.0.log Journal 200 recovered. -117> 2014-08-20 06:12:09.503476 7f32f8608700 4 mds.0.log Recovered journal 200 in format 1 -116> 2014-08-20 06:12:09.503483 7f32f8608700 2 mds.0.4 boot_start 1: loading/discovering base inodes -115> 2014-08-20 06:12:09.503491 7f32f8608700 0 mds.0.cache creating system inode with ino:100 -114> 2014-08-20 06:12:09.503601 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:8 100.00000000 [getxattr inode] 1.c5265ab3 ack+read+known_if_redirected e10) v4 -- ?+0 0x2d8bc80 con 0x2d729a0 -113> 2014-08-20 06:12:09.503647 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:9 100.00000000.inode [read 0~0] 1.85dde07f ack+read+known_if_redirected e10) v4 -- ?+0 0x2da9000 con 0x2d729a0 -112> 2014-08-20 06:12:09.503669 7f32f8608700 0 mds.0.cache creating system inode with ino:1 -111> 2014-08-20 06:12:09.503700 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6805/29464 -- osd_op(mds.0.4:10 1.00000000 [getxattr inode] 1.6b2cdaff ack+read+known_if_redirected e10) v4 -- ?+0 0x2da9280 con 0x2d72840 -110> 2014-08-20 06:12:09.503737 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6805/29464 -- osd_op(mds.0.4:11 1.00000000.inode [read 0~0] 1.232c0e14 ack+read+known_if_redirected e10) v4 -- ?+0 0x2da9780 con 0x2d72840 -109> 2014-08-20 06:12:09.504823 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 6 ==== osd_op_reply(8 100.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v6 ==== 179+0+0 (597178 0 0) 0x2d80b80 con 0x2d729a0 -108> 2014-08-20 06:12:09.504881 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.2 10.214.137.130:6805/29464 2 ==== osd_op_reply(10 1.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v6 ==== 177+0+0 (2877742614 0 0) 0x2db4b00 con 0x2d72840 -107> 2014-08-20 06:12:09.504916 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 7 ==== osd_op_reply(9 100.00000000.inode [read 0~452] v0'0 uv1 ondisk = 0) v6 ==== 185+0+452 (3460037969 0 3945391326) 0x2db5600 con 0x2d729a0 -106> 2014-08-20 06:12:09.504972 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.2 10.214.137.130:6805/29464 3 ==== osd_op_reply(11 1.00000000.inode [read 0~452] v0'0 uv1 ondisk = 0) v6 ==== 183+0+452 (3377856095 0 4079482348) 0x2db4840 con 0x2d72840 -105> 2014-08-20 06:12:09.505003 7f32fc112700 2 mds.0.4 boot_start 2: replaying mds log -104> 2014-08-20 06:12:09.505086 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:12 200.00000001 [read 0~20168] 1.6e5f474 ack+read+known_if_redirected e10) v4 -- ?+0 0x2da9500 con 0x2d729a0 -103> 2014-08-20 06:12:09.506241 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 8 ==== osd_op_reply(12 200.00000001 [read 0~20168] v0'0 uv24 ondisk = 0) v6 ==== 179+0+20168 (971954365 0 2012295194) 0x2db5340 con 0x2d729a0 -102> 2014-08-20 06:12:09.506972 7f32f8608700 1 mds.0.4 replay_done -101> 2014-08-20 06:12:09.506986 7f32f8608700 1 mds.0.4 making mds journal writeable -100> 2014-08-20 06:12:09.506990 7f32f8608700 1 mds.0.journaler(ro) set_writeable -99> 2014-08-20 06:12:09.507032 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:13 200.00000001 [zero 20168~4174136] 1.6e5f474 ondisk+write+known_if_redirected e10) v4 -- ?+0 0x2da9280 con 0x2d729a0 -98> 2014-08-20 06:12:09.507103 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:14 200.00000002 [delete] 1.eb272dbb ondisk+write+known_if_redirected e10) v4 -- ?+0 0x2da9c80 con 0x2d729a0 -97> 2014-08-20 06:12:09.507155 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6805/29464 -- osd_op(mds.0.4:15 200.00000003 [delete] 1.95e3ab ondisk+write+known_if_redirected e10) v4 -- ?+0 0x2dd8780 con 0x2d72840 -96> 2014-08-20 06:12:09.507209 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6800/28669 -- osd_op(mds.0.4:16 200.00000004 [delete] 1.1ba6bc1f ondisk+write+known_if_redirected e10) v4 -- ?+0 0x2dd8500 con 0x2d726e0 -95> 2014-08-20 06:12:09.507258 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6800/28669 -- osd_op(mds.0.4:17 200.00000005 [delete] 1.96d5827a ondisk+write+known_if_redirected e10) v4 -- ?+0 0x2dd8280 con 0x2d726e0 -94> 2014-08-20 06:12:09.507380 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:18 200.00000006 [delete] 1.ade7efde ondisk+write+known_if_redirected e10) v4 -- ?+0 0x2dd8000 con 0x2d729a0 -93> 2014-08-20 06:12:09.507409 7f32f8608700 2 mds.0.4 i am alone, moving to state reconnect -92> 2014-08-20 06:12:09.507418 7f32f8608700 3 mds.0.4 request_state up:reconnect -91> 2014-08-20 06:12:09.507441 7f32f8608700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -90> 2014-08-20 06:12:09.507453 7f32f8608700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 2 v18) v2 -- ?+0 0x2d80b80 con 0x2d722c0 -89> 2014-08-20 06:12:09.508287 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 9 ==== osd_op_reply(13 200.00000001 [zero 20168~4174136] v0'0 uv24 ondisk = 0) v6 ==== 179+0+0 (4274359799 0 0) 0x2db5080 con 0x2d729a0 -88> 2014-08-20 06:12:09.508339 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.2 10.214.137.130:6805/29464 4 ==== osd_op_reply(15 200.00000003 [delete] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v6 ==== 179+0+0 (864825720 0 0) 0x2db4580 con 0x2d72840 -87> 2014-08-20 06:12:09.508379 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.0 10.214.136.128:6800/28669 2 ==== osd_op_reply(16 200.00000004 [delete] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v6 ==== 179+0+0 (510520897 0 0) 0x2db4000 con 0x2d726e0 -86> 2014-08-20 06:12:09.508492 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.0 10.214.136.128:6800/28669 3 ==== osd_op_reply(17 200.00000005 [delete] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v6 ==== 179+0+0 (2898014583 0 0) 0x2db42c0 con 0x2d726e0 -85> 2014-08-20 06:12:09.508544 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 10 ==== osd_op_reply(14 200.00000002 [delete] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v6 ==== 179+0+0 (1968108432 0 0) 0x2db4dc0 con 0x2d729a0 -84> 2014-08-20 06:12:09.508610 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 11 ==== osd_op_reply(18 200.00000006 [delete] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v6 ==== 179+0+0 (3210369299 0 0) 0x2db4580 con 0x2d729a0 -83> 2014-08-20 06:12:10.495240 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 17 ==== mdsmap(e 19) v1 ==== 661+0+0 (3106833483 0 0) 0x2da5480 con 0x2d722c0 -82> 2014-08-20 06:12:10.495271 7f32fc112700 5 mds.0.4 handle_mds_map epoch 19 from mon.0 -81> 2014-08-20 06:12:10.495302 7f32fc112700 1 mds.0.4 handle_mds_map i am now mds.0.4 -80> 2014-08-20 06:12:10.495309 7f32fc112700 1 mds.0.4 handle_mds_map state change up:replay --> up:reconnect -79> 2014-08-20 06:12:10.495312 7f32fc112700 1 mds.0.4 reconnect_start -78> 2014-08-20 06:12:10.495314 7f32fc112700 1 mds.0.4 reopen_log -77> 2014-08-20 06:12:10.495322 7f32fc112700 1 mds.0.server reconnect_clients -- 1 sessions -76> 2014-08-20 06:12:10.495361 7f32fc112700 5 mds.0.bal rebalance done -75> 2014-08-20 06:12:10.496160 7f32f7202700 1 -- 10.214.136.128:6805/29505 >> :/0 pipe(0x2d80340 sd=20 :6805 s=0 pgs=0 cs=0 l=0 c=0x2d72420).accept sd=20 10.214.137.130:43705/0 -74> 2014-08-20 06:12:10.497584 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== client.4126 10.214.137.130:0/29908 1 ==== client_reconnect(0 caps) v3 ==== 0+0+4 (0 0 0) 0x2d92e00 con 0x2d72420 -73> 2014-08-20 06:12:10.497631 7f32fc112700 1 mds.0.server session is closed, ignoring reconnect, sending close -72> 2014-08-20 06:12:10.497667 7f32fc112700 0 log [INF] : denied reconnect attempt (mds is up:reconnect) from client.4126 10.214.137.130:0/29908 (session is closed) -71> 2014-08-20 06:12:10.497677 7f32fc112700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:0/29908 -- client_session(close) v1 -- ?+0 0x2d92a00 con 0x2d72420 -70> 2014-08-20 06:12:10.498375 7f32f7202700 2 -- 10.214.136.128:6805/29505 >> 10.214.137.130:0/29908 pipe(0x2d80340 sd=20 :6805 s=2 pgs=4 cs=1 l=0 c=0x2d72420).reader couldn't read tag, (0) Success -69> 2014-08-20 06:12:10.498416 7f32f7202700 2 -- 10.214.136.128:6805/29505 >> 10.214.137.130:0/29908 pipe(0x2d80340 sd=20 :6805 s=2 pgs=4 cs=1 l=0 c=0x2d72420).fault (0) Success -68> 2014-08-20 06:12:10.498447 7f32f7202700 0 -- 10.214.136.128:6805/29505 >> 10.214.137.130:0/29908 pipe(0x2d80340 sd=20 :6805 s=2 pgs=4 cs=1 l=0 c=0x2d72420).fault, server, going to standby -67> 2014-08-20 06:12:11.478269 7f32fa90f700 10 monclient: tick -66> 2014-08-20 06:12:11.478299 7f32fa90f700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-08-20 06:11:41.478297) -65> 2014-08-20 06:12:11.478313 7f32fa90f700 10 monclient: renew subs? (now: 2014-08-20 06:12:11.478313; renew after: 2014-08-20 06:14:39.492491) -- no -64> 2014-08-20 06:12:11.478327 7f32fa90f700 10 log_queue is 1 last_log 1 sent 0 num 1 unsent 1 sending 1 -63> 2014-08-20 06:12:11.478340 7f32fa90f700 10 will send 2014-08-20 06:12:10.497671 mds.0 10.214.136.128:6805/29505 1 : [INF] denied reconnect attempt (mds is up:reconnect) from client.4126 10.214.137.130:0/29908 (session is closed) -62> 2014-08-20 06:12:11.478370 7f32fa90f700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -61> 2014-08-20 06:12:11.478382 7f32fa90f700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- log(1 entries) v1 -- ?+0 0x2dd1900 con 0x2d722c0 -60> 2014-08-20 06:12:11.496824 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 18 ==== log(last 1) v1 ==== 24+0+0 (3012353504 0 0) 0x2d9d700 con 0x2d722c0 -59> 2014-08-20 06:12:11.496862 7f32fc112700 10 handle_log_ack log(last 1) v1 -58> 2014-08-20 06:12:11.496867 7f32fc112700 10 logged 2014-08-20 06:12:10.497671 mds.0 10.214.136.128:6805/29505 1 : [INF] denied reconnect attempt (mds is up:reconnect) from client.4126 10.214.137.130:0/29908 (session is closed) -57> 2014-08-20 06:12:13.507597 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -56> 2014-08-20 06:12:13.507614 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 3 v19) v2 -- ?+0 0x2db5b80 con 0x2d722c0 -55> 2014-08-20 06:12:13.514967 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 19 ==== mdsbeacon(4125/a up:reconnect seq 3 v19) v2 ==== 103+0+0 (1785179744 0 0) 0x2db4b00 con 0x2d722c0 -54> 2014-08-20 06:12:17.507843 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -53> 2014-08-20 06:12:17.507863 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 4 v19) v2 -- ?+0 0x2dee2c0 con 0x2d722c0 -52> 2014-08-20 06:12:17.508675 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 20 ==== mdsbeacon(4125/a up:reconnect seq 4 v19) v2 ==== 103+0+0 (5614244 0 0) 0x2db4000 con 0x2d722c0 -51> 2014-08-20 06:12:21.478587 7f32fa90f700 10 monclient: tick -50> 2014-08-20 06:12:21.478606 7f32fa90f700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-08-20 06:11:51.478605) -49> 2014-08-20 06:12:21.478619 7f32fa90f700 10 monclient: renew subs? (now: 2014-08-20 06:12:21.478619; renew after: 2014-08-20 06:14:39.492491) -- no -48> 2014-08-20 06:12:21.508084 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -47> 2014-08-20 06:12:21.508104 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 5 v19) v2 -- ?+0 0x2dee000 con 0x2d722c0 -46> 2014-08-20 06:12:21.508902 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 21 ==== mdsbeacon(4125/a up:reconnect seq 5 v19) v2 ==== 103+0+0 (2038224835 0 0) 0x2db5b80 con 0x2d722c0 -45> 2014-08-20 06:12:25.508319 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -44> 2014-08-20 06:12:25.508342 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 6 v19) v2 -- ?+0 0x2db4580 con 0x2d722c0 -43> 2014-08-20 06:12:25.509156 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 22 ==== mdsbeacon(4125/a up:reconnect seq 6 v19) v2 ==== 103+0+0 (4060565610 0 0) 0x2db5b80 con 0x2d722c0 -42> 2014-08-20 06:12:29.508563 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -41> 2014-08-20 06:12:29.508583 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 7 v19) v2 -- ?+0 0x2db4dc0 con 0x2d722c0 -40> 2014-08-20 06:12:29.509395 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 23 ==== mdsbeacon(4125/a up:reconnect seq 7 v19) v2 ==== 103+0+0 (2335059213 0 0) 0x2db4000 con 0x2d722c0 -39> 2014-08-20 06:12:31.478822 7f32fa90f700 10 monclient: tick -38> 2014-08-20 06:12:31.478840 7f32fa90f700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-08-20 06:12:01.478839) -37> 2014-08-20 06:12:31.478850 7f32fa90f700 10 monclient: renew subs? (now: 2014-08-20 06:12:31.478850; renew after: 2014-08-20 06:14:39.492491) -- no -36> 2014-08-20 06:12:33.508714 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -35> 2014-08-20 06:12:33.508734 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 8 v19) v2 -- ?+0 0x2db42c0 con 0x2d722c0 -34> 2014-08-20 06:12:33.509550 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 24 ==== mdsbeacon(4125/a up:reconnect seq 8 v19) v2 ==== 103+0+0 (644045282 0 0) 0x2db4b00 con 0x2d722c0 -33> 2014-08-20 06:12:37.508941 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -32> 2014-08-20 06:12:37.508961 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 9 v19) v2 -- ?+0 0x2df0840 con 0x2d722c0 -31> 2014-08-20 06:12:37.509744 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 25 ==== mdsbeacon(4125/a up:reconnect seq 9 v19) v2 ==== 103+0+0 (1598694533 0 0) 0x2db58c0 con 0x2d722c0 -30> 2014-08-20 06:12:41.479052 7f32fa90f700 10 monclient: tick -29> 2014-08-20 06:12:41.479069 7f32fa90f700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-08-20 06:12:11.479068) -28> 2014-08-20 06:12:41.479080 7f32fa90f700 10 monclient: renew subs? (now: 2014-08-20 06:12:41.479080; renew after: 2014-08-20 06:14:39.492491) -- no -27> 2014-08-20 06:12:41.509164 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -26> 2014-08-20 06:12:41.509183 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 10 v19) v2 -- ?+0 0x2df0580 con 0x2d722c0 -25> 2014-08-20 06:12:41.509976 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 26 ==== mdsbeacon(4125/a up:reconnect seq 10 v19) v2 ==== 103+0+0 (3560022828 0 0) 0x2df0dc0 con 0x2d722c0 -24> 2014-08-20 06:12:45.509401 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -23> 2014-08-20 06:12:45.509421 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 11 v19) v2 -- ?+0 0x2df02c0 con 0x2d722c0 -22> 2014-08-20 06:12:45.510229 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 27 ==== mdsbeacon(4125/a up:reconnect seq 11 v19) v2 ==== 103+0+0 (2904088139 0 0) 0x2df0b00 con 0x2d722c0 -21> 2014-08-20 06:12:49.509674 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -20> 2014-08-20 06:12:49.509692 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 12 v19) v2 -- ?+0 0x2df0000 con 0x2d722c0 -19> 2014-08-20 06:12:49.510464 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 28 ==== mdsbeacon(4125/a up:reconnect seq 12 v19) v2 ==== 103+0+0 (3341474447 0 0) 0x2df0840 con 0x2d722c0 -18> 2014-08-20 06:12:51.479283 7f32fa90f700 10 monclient: tick -17> 2014-08-20 06:12:51.479300 7f32fa90f700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2014-08-20 06:12:21.479299) -16> 2014-08-20 06:12:51.479311 7f32fa90f700 10 monclient: renew subs? (now: 2014-08-20 06:12:51.479311; renew after: 2014-08-20 06:14:39.492491) -- no -15> 2014-08-20 06:12:53.509901 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -14> 2014-08-20 06:12:53.509921 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 13 v19) v2 -- ?+0 0x2df1b80 con 0x2d722c0 -13> 2014-08-20 06:12:53.510625 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 29 ==== mdsbeacon(4125/a up:reconnect seq 13 v19) v2 ==== 103+0+0 (3187910632 0 0) 0x2db42c0 con 0x2d722c0 -12> 2014-08-20 06:12:57.510140 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -11> 2014-08-20 06:12:57.510160 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:reconnect seq 14 v19) v2 -- ?+0 0x2df18c0 con 0x2d722c0 -10> 2014-08-20 06:12:57.510925 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== mon.0 10.214.136.128:6789/0 30 ==== mdsbeacon(4125/a up:reconnect seq 14 v19) v2 ==== 103+0+0 (897065025 0 0) 0x2db4dc0 con 0x2d722c0 -9> 2014-08-20 06:12:58.485365 7f32f980c700 1 mds.0.server reconnect gave up on client.4123 10.214.137.130:0/29770 -8> 2014-08-20 06:12:58.485403 7f32f980c700 1 mds.0.4 reconnect_done -7> 2014-08-20 06:12:58.485407 7f32f980c700 3 mds.0.4 request_state up:rejoin -6> 2014-08-20 06:12:58.485417 7f32f980c700 10 monclient: _send_mon_message to mon.a at 10.214.136.128:6789/0 -5> 2014-08-20 06:12:58.485428 7f32f980c700 1 -- 10.214.136.128:6805/29505 --> 10.214.136.128:6789/0 -- mdsbeacon(4125/a up:rejoin seq 15 v19) v2 -- ?+0 0x2df1600 con 0x2d722c0 -4> 2014-08-20 06:12:58.485557 7f32f7e07700 5 mds.0.log _submit_thread 4214472~194 : ESession client.4123 10.214.137.130:0/29770 close cmapv 16 -3> 2014-08-20 06:12:58.485660 7f32f7e07700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:19 200.00000001 [write 20168~214] 1.6e5f474 ondisk+write+known_if_redirected e10) v4 -- ?+0 0x2dd8500 con 0x2d729a0 -2> 2014-08-20 06:12:58.485728 7f32f7e07700 1 -- 10.214.136.128:6805/29505 --> 10.214.137.130:6800/29383 -- osd_op(mds.0.4:20 200.00000000 [writefull 0~90] 1.844f3494 ondisk+write+known_if_redirected e10) v4 -- ?+0 0x2dd8c80 con 0x2d729a0 -1> 2014-08-20 06:12:58.490615 7f32fc112700 1 -- 10.214.136.128:6805/29505 <== osd.1 10.214.137.130:6800/29383 12 ==== osd_op_reply(19 200.00000001 [write 20168~214] v10'25 uv25 ondisk = 0) v6 ==== 179+0+0 (3483442449 0 0) 0x2db5080 con 0x2d729a0 0> 2014-08-20 06:12:58.495556 7f32fc112700 -1 *** Caught signal (Aborted) ** in thread 7f32fc112700 ceph version 0.84-365-g790de97 (790de974a88c1cd26ca5b96d85b6ed8a5a5bfa8b) 1: ceph-mds() [0x810cfa] 2: (()+0xfcb0) [0x7f3300a01cb0] 3: (gsignal()+0x35) [0x7f32ff4f0425] 4: (abort()+0x17b) [0x7f32ff4f3b8b] 5: (()+0x2f0ee) [0x7f32ff4e90ee] 6: (()+0x2f192) [0x7f32ff4e9192] 7: ceph-mds() [0x57215d] 8: (Server::_session_logged(Session*, unsigned long, bool, unsigned long, interval_set&, unsigned long)+0x151b) [0x5adceb] 9: (C_MDS_session_finish::finish(int)+0x38) [0x5ff478] 10: (Context::complete(int)+0x9) [0x595859] 11: (finish_contexts(CephContext*, std::list >&, int)+0x95) [0x597065] 12: (Journaler::_finish_flush(int, unsigned long, utime_t)+0x1b2) [0x7a3b82] 13: (Context::complete(int)+0x9) [0x595859] 14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x1073) [0x7be213] 15: (MDS::handle_core_message(Message*)+0x5b8) [0x592948] 16: (MDS::_dispatch(Message*)+0x2f) [0x59308f] 17: (MDS::ms_dispatch(Message*)+0x1e4) [0x594af4] 18: (Messenger::ms_deliver_dispatch(Message*)+0x77) [0x9c3f87] 19: (DispatchQueue::entry()+0x4b2) [0x9c0d22] 20: (DispatchQueue::DispatchThread::entry()+0xd) [0x8d5bcd] 21: (()+0x7e9a) [0x7f33009f9e9a] 22: (clone()+0x6d) [0x7f32ff5ae3fd] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.a.log --- end dump of recent events ---