2016-08-26 09:32:45.970573 7fd650372180 0 set uid:gid to 1004:1001 (ceph:ceph) 2016-08-26 09:32:45.970596 7fd650372180 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mds, pid 2926836 2016-08-26 09:32:45.970901 7fd650372180 0 pidfile_write: ignore empty --pid-file 2016-08-26 09:32:46.115577 7fd64900f700 1 mds.rack3-storage-7 handle_mds_map standby 2016-08-26 09:32:46.118134 7fd64900f700 1 mds.0.24 handle_mds_map i am now mds.0.24 2016-08-26 09:32:46.118152 7fd64900f700 1 mds.0.24 handle_mds_map state change up:boot --> up:replay 2016-08-26 09:32:46.118156 7fd64900f700 1 mds.0.24 replay_start 2016-08-26 09:32:46.118159 7fd64900f700 1 mds.0.24 recovery set is 2016-08-26 09:32:46.122323 7fd643e03700 0 mds.0.cache creating system inode with ino:100 2016-08-26 09:32:46.122426 7fd643e03700 0 mds.0.cache creating system inode with ino:1 2016-08-26 09:32:47.507828 7fd6421f5700 1 mds.0.24 replay_done 2016-08-26 09:32:47.507843 7fd6421f5700 1 mds.0.24 making mds journal writeable 2016-08-26 09:32:48.123512 7fd64900f700 1 mds.0.24 handle_mds_map i am now mds.0.24 2016-08-26 09:32:48.123516 7fd64900f700 1 mds.0.24 handle_mds_map state change up:replay --> up:reconnect 2016-08-26 09:32:48.123522 7fd64900f700 1 mds.0.24 reconnect_start 2016-08-26 09:32:48.123525 7fd64900f700 1 mds.0.24 reopen_log 2016-08-26 09:32:48.123534 7fd64900f700 1 mds.0.server reconnect_clients -- 3 sessions 2016-08-26 09:32:48.123807 7fd6407e2700 0 -- 10.242.43.21:6800/2926836 >> 10.242.43.91:0/733588789 pipe(0x7fd65aea2000 sd=33 :6800 s=0 pgs=0 cs=0 l=0 c=0x7fd65ad55780).accept peer addr is really 10.242.43.91:0/733588789 (socket is 10.242.43.91:44002/0) 2016-08-26 09:32:48.123941 7fd6407e2700 1 mds.client.guest ms_verify_authorizer: cannot decode auth caps bl of length 0 2016-08-26 09:32:48.124054 7fd6406e1700 0 -- 10.242.43.21:6800/2926836 >> 10.242.43.89:0/3139122963 pipe(0x7fd65aea3400 sd=34 :6800 s=0 pgs=0 cs=0 l=0 c=0x7fd65ae96480).accept peer addr is really 10.242.43.89:0/3139122963 (socket is 10.242.43.89:36523/0) 2016-08-26 09:32:48.124156 7fd64900f700 0 log_channel(cluster) log [DBG] : reconnect by client.15950 10.242.43.91:0/733588789 after 0.000584 2016-08-26 09:32:48.124256 7fd6406e1700 1 mds.client.guest ms_verify_authorizer: cannot decode auth caps bl of length 0 2016-08-26 09:32:48.124384 7fd64900f700 0 log_channel(cluster) log [DBG] : reconnect by client.15949 10.242.43.89:0/3139122963 after 0.000849 2016-08-26 09:33:36.007542 7fd64680a700 1 mds.0.server reconnect gave up on client.15414 10.242.43.127:0/3383760195 2016-08-26 09:33:36.007586 7fd64680a700 1 mds.0.24 reconnect_done 2016-08-26 09:33:36.060966 7fd64900f700 1 mds.0.24 handle_mds_map i am now mds.0.24 2016-08-26 09:33:36.060969 7fd64900f700 1 mds.0.24 handle_mds_map state change up:reconnect --> up:rejoin 2016-08-26 09:33:36.060977 7fd64900f700 1 mds.0.24 rejoin_start 2016-08-26 09:33:36.061294 7fd64900f700 1 mds.0.24 rejoin_joint_start 2016-08-26 09:33:36.063929 7fd645007700 1 mds.0.24 rejoin_done 2016-08-26 09:33:37.066912 7fd64900f700 1 mds.0.24 handle_mds_map i am now mds.0.24 2016-08-26 09:33:37.066915 7fd64900f700 1 mds.0.24 handle_mds_map state change up:rejoin --> up:active 2016-08-26 09:33:37.066920 7fd64900f700 1 mds.0.24 recovery_done -- successful recovery! 2016-08-26 09:33:37.067055 7fd64900f700 1 mds.0.24 active_start 2016-08-26 09:33:37.067163 7fd64900f700 1 mds.0.24 cluster recovered. 2016-08-26 09:33:43.246663 7fd6403de700 1 mds.client.guest ms_verify_authorizer: cannot decode auth caps bl of length 0 2016-08-26 09:34:56.008534 7fd64680a700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 33.487021 secs 2016-08-26 09:34:56.008547 7fd64680a700 0 log_channel(cluster) log [WRN] : slow request 33.487021 seconds old, received at 2016-08-26 09:34:22.521443: client_request(client.233987:6 getattr pAsLsXsFs #100000f6966 2016-08-26 09:34:22.492964) currently failed to rdlock, waiting 2016-08-26 09:35:26.008884 7fd64680a700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 63.487383 secs 2016-08-26 09:35:26.008894 7fd64680a700 0 log_channel(cluster) log [WRN] : slow request 63.487383 seconds old, received at 2016-08-26 09:34:22.521443: client_request(client.233987:6 getattr pAsLsXsFs #100000f6966 2016-08-26 09:34:22.492964) currently failed to rdlock, waiting 2016-08-26 09:36:26.009625 7fd64680a700 0 log_channel(cluster) log [WRN] : 1 slow requests, 1 included below; oldest blocked for > 123.488102 secs 2016-08-26 09:36:26.009636 7fd64680a700 0 log_channel(cluster) log [WRN] : slow request 123.488102 seconds old, received at 2016-08-26 09:34:22.521443: client_request(client.233987:6 getattr pAsLsXsFs #100000f6966 2016-08-26 09:34:22.492964) currently failed to rdlock, waiting 2016-08-26 09:36:41.009771 7fd64680a700 0 log_channel(cluster) log [WRN] : 2 slow requests, 1 included below; oldest blocked for > 138.488281 secs 2016-08-26 09:36:41.009778 7fd64680a700 0 log_channel(cluster) log [WRN] : slow request 32.174536 seconds old, received at 2016-08-26 09:36:08.835189: client_request(client.233987:9 getattr pAsLsXsFs #100000f6966 2016-08-26 09:36:08.804964) currently failed to rdlock, waiting 2016-08-26 09:37:11.010166 7fd64680a700 0 log_channel(cluster) log [WRN] : 2 slow requests, 1 included below; oldest blocked for > 168.488650 secs 2016-08-26 09:37:11.010176 7fd64680a700 0 log_channel(cluster) log [WRN] : slow request 62.174905 seconds old, received at 2016-08-26 09:36:08.835189: client_request(client.233987:9 getattr pAsLsXsFs #100000f6966 2016-08-26 09:36:08.804964) currently failed to rdlock, waiting 2016-08-26 09:49:27.639665 7fd645808700 -1 mds.rack3-storage-7 *** got signal Terminated *** 2016-08-26 09:49:27.639678 7fd645808700 1 mds.rack3-storage-7 suicide. wanted state up:active 2016-08-26 09:49:27.642683 7fd645808700 1 mds.0.24 shutdown: shutting down rank 0 2016-08-26 09:54:12.646899 7f96911e2180 0 set uid:gid to 1004:1001 (ceph:ceph) 2016-08-26 09:54:12.646927 7f96911e2180 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mds, pid 2939808 2016-08-26 09:54:12.647237 7f96911e2180 0 pidfile_write: ignore empty --pid-file 2016-08-26 09:54:12.736910 7f9689e7f700 1 mds.rack3-storage-7 handle_mds_map standby 2016-08-26 09:54:12.739571 7f9689e7f700 1 mds.0.30 handle_mds_map i am now mds.0.30 2016-08-26 09:54:12.739579 7f9689e7f700 1 mds.0.30 handle_mds_map state change up:boot --> up:replay 2016-08-26 09:54:12.739583 7f9689e7f700 1 mds.0.30 replay_start 2016-08-26 09:54:12.739585 7f9689e7f700 1 mds.0.30 recovery set is 2016-08-26 09:54:12.744317 7f9684c73700 0 mds.0.cache creating system inode with ino:100 2016-08-26 09:54:12.744401 7f9684c73700 0 mds.0.cache creating system inode with ino:1 2016-08-26 09:54:14.165544 7f9683065700 1 mds.0.30 replay_done 2016-08-26 09:54:14.165574 7f9683065700 1 mds.0.30 making mds journal writeable 2016-08-26 09:54:14.744368 7f9689e7f700 1 mds.0.30 handle_mds_map i am now mds.0.30 2016-08-26 09:54:14.744371 7f9689e7f700 1 mds.0.30 handle_mds_map state change up:replay --> up:reconnect 2016-08-26 09:54:14.744377 7f9689e7f700 1 mds.0.30 reconnect_start 2016-08-26 09:54:14.744380 7f9689e7f700 1 mds.0.30 reopen_log 2016-08-26 09:54:14.744383 7f9689e7f700 1 mds.0.server reconnect_clients -- 3 sessions 2016-08-26 09:54:14.744652 7f9681652700 0 -- 10.242.43.21:6800/2939808 >> 10.242.43.91:0/733588789 pipe(0x7f969ac08000 sd=33 :6800 s=0 pgs=0 cs=0 l=0 c=0x7f969aac7780).accept peer addr is really 10.242.43.91:0/733588789 (socket is 10.242.43.91:44003/0) 2016-08-26 09:54:14.744784 7f9681652700 1 mds.client.guest ms_verify_authorizer: cannot decode auth caps bl of length 0 2016-08-26 09:54:14.744863 7f9681551700 1 mds.client.guest ms_verify_authorizer: cannot decode auth caps bl of length 0 2016-08-26 09:54:14.744967 7f9689e7f700 0 log_channel(cluster) log [DBG] : reconnect by client.15950 10.242.43.91:0/733588789 after 0.000531 2016-08-26 09:54:14.745039 7f9689e7f700 0 log_channel(cluster) log [DBG] : reconnect by client.233987 10.242.43.127:0/775503929 after 0.000647 2016-08-26 09:55:02.683677 7f968767a700 1 mds.0.server reconnect gave up on client.15949 10.242.43.89:0/3139122963 2016-08-26 09:55:02.683726 7f968767a700 1 mds.0.30 reconnect_done 2016-08-26 09:55:03.108858 7f9689e7f700 1 mds.0.30 handle_mds_map i am now mds.0.30 2016-08-26 09:55:03.108863 7f9689e7f700 1 mds.0.30 handle_mds_map state change up:reconnect --> up:rejoin 2016-08-26 09:55:03.108869 7f9689e7f700 1 mds.0.30 rejoin_start 2016-08-26 09:55:03.109036 7f9689e7f700 1 mds.0.30 rejoin_joint_start 2016-08-26 09:55:03.116191 7f9685e77700 1 mds.0.30 rejoin_done 2016-08-26 09:55:04.114361 7f9689e7f700 1 mds.0.30 handle_mds_map i am now mds.0.30 2016-08-26 09:55:04.114364 7f9689e7f700 1 mds.0.30 handle_mds_map state change up:rejoin --> up:active 2016-08-26 09:55:04.114370 7f9689e7f700 1 mds.0.30 recovery_done -- successful recovery! 2016-08-26 09:55:04.114428 7f9689e7f700 1 mds.0.30 active_start 2016-08-26 09:55:04.114537 7f9689e7f700 1 mds.0.30 cluster recovered. 2016-08-26 09:56:36.502628 7f968124e700 1 mds.client.guest ms_verify_authorizer: cannot decode auth caps bl of length 0