Project

General

Profile

Bug #4434 ยป ceph-mds.a.log

Ken Franklin, 03/13/2013 02:36 PM

 
2013-03-13 06:29:48.609420 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6804/42117
2013-03-13 06:29:48.609777 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6804/42117
2013-03-13 06:29:52.784861 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6801/41910
2013-03-13 06:29:52.785196 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6801/41910
2013-03-13 06:44:48.709971 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6804/42117
2013-03-13 06:44:48.710448 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6804/42117
2013-03-13 06:44:52.884292 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6801/41910
2013-03-13 06:44:52.884652 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6801/41910
2013-03-13 06:59:48.774666 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6804/42117
2013-03-13 06:59:48.775054 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6804/42117
2013-03-13 06:59:52.901394 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6801/41910
2013-03-13 06:59:52.901990 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6801/41910
2013-03-13 07:14:48.800124 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6804/42117
2013-03-13 07:14:48.800457 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6804/42117
2013-03-13 07:14:52.993933 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6801/41910
2013-03-13 07:14:52.994288 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6801/41910
2013-03-13 07:29:48.801654 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6804/42117
2013-03-13 07:29:48.802046 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6804/42117
2013-03-13 07:29:53.094419 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6801/41910
2013-03-13 07:29:53.094797 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6801/41910
2013-03-13 07:44:48.830747 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6804/42117
2013-03-13 07:44:48.831188 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6804/42117
2013-03-13 07:44:53.132559 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6801/41910
2013-03-13 07:44:53.132902 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6801/41910
2013-03-13 07:59:48.916402 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6804/42117
2013-03-13 07:59:48.916766 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6804/42117
2013-03-13 07:59:53.141207 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6801/41910
2013-03-13 07:59:53.141677 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6801/41910
2013-03-13 08:10:04.755549 7fef2d262700 0 monclient: hunting for new mon
2013-03-13 08:10:04.755627 7fef2d262700 0 mds.0.18 ms_handle_reset on 10.214.136.10:6789/0
2013-03-13 08:10:04.756071 7fef2a85b700 0 -- 10.214.136.10:6800/41692 >> 10.214.136.10:6789/0 pipe(0x561d280 sd=21 :43961 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 08:10:06.157394 7fef2d262700 0 mds.0.18 ms_handle_connect on 10.214.136.10:6789/0
2013-03-13 08:10:06.173052 7fef2da63700 -1 mds.0.18 *** got signal Terminated ***
2013-03-13 08:10:06.173070 7fef2da63700 1 mds.0.18 suicide. wanted down:dne, now up:active
2013-03-13 08:10:06.238225 7fef327b5780 0 stopped.
2013-03-13 08:10:07.378352 7f556e3a8700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
2013-03-13 08:10:07.804145 7f556e3a8700 1 mds.-1.0 handle_mds_map standby
2013-03-13 08:10:23.966791 7f556e3a8700 1 mds.0.19 handle_mds_map i am now mds.0.19
2013-03-13 08:10:23.966793 7f556e3a8700 1 mds.0.19 handle_mds_map state change up:standby --> up:replay
2013-03-13 08:10:23.966799 7f556e3a8700 1 mds.0.19 replay_start
2013-03-13 08:10:23.966803 7f556e3a8700 1 mds.0.19 recovery set is
2013-03-13 08:10:23.966804 7f556e3a8700 1 mds.0.19 need osdmap epoch 542, have 541
2013-03-13 08:10:23.966807 7f556e3a8700 1 mds.0.19 waiting for osdmap 542 (which blacklists prior instance)
2013-03-13 08:10:23.966836 7f556e3a8700 1 mds.0.cache handle_mds_failure mds.0 : recovery peers are
2013-03-13 08:10:24.360695 7f55738f9700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6801/41910 pipe(0x3230a00 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 08:10:24.360777 7f556b9a1700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6804/42117 pipe(0x3230c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 08:10:37.632959 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6802/55225
2013-03-13 08:10:42.845637 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6805/55437
2013-03-13 08:10:56.103505 7f556e3a8700 0 mds.0.cache creating system inode with ino:100
2013-03-13 08:10:56.103675 7f556e3a8700 0 mds.0.cache creating system inode with ino:1
2013-03-13 08:10:57.992944 7f556b69e700 1 mds.0.19 replay_done
2013-03-13 08:10:57.992974 7f556b69e700 1 mds.0.19 making mds journal writeable
2013-03-13 08:11:01.549041 7f556e3a8700 1 mds.0.19 handle_mds_map i am now mds.0.19
2013-03-13 08:11:01.549045 7f556e3a8700 1 mds.0.19 handle_mds_map state change up:replay --> up:reconnect
2013-03-13 08:11:01.549057 7f556e3a8700 1 mds.0.19 reconnect_start
2013-03-13 08:11:01.549059 7f556e3a8700 1 mds.0.19 reopen_log
2013-03-13 08:11:01.549063 7f556e3a8700 1 mds.0.server reconnect_clients -- 1 sessions
2013-03-13 08:11:01.549710 7f556e3a8700 0 log [DBG] : reconnect by client.8302 10.214.136.10:0/43438 after 0.000607
2013-03-13 08:11:01.549738 7f556e3a8700 0 mds.0.server missing 10000000000 #1/iogen_23783.io (mine), will load later
2013-03-13 08:11:01.549748 7f556e3a8700 0 mds.0.server missing 10000005b29 #1/iogen_38485.io (mine), will load later
2013-03-13 08:11:01.549751 7f556e3a8700 0 mds.0.server missing 1000000b6f0 #1/iogen_47410.io (mine), will load later
2013-03-13 08:11:01.549754 7f556e3a8700 0 mds.0.server missing 1000001135d #1/iogen_56598.io (mine), will load later
2013-03-13 08:11:01.549758 7f556e3a8700 0 mds.0.server missing 10000016f4a #1/iogen_633.io (mine), will load later
2013-03-13 08:11:01.549760 7f556e3a8700 0 mds.0.server missing 1000001ca7c #1/iogen_12694.io (mine), will load later
2013-03-13 08:11:01.549763 7f556e3a8700 0 mds.0.server missing 10000022515 #1/iogen_27233.io (mine), will load later
2013-03-13 08:11:01.549766 7f556e3a8700 0 mds.0.server missing 1000002813a #1/iogen_35766.io (mine), will load later
2013-03-13 08:11:01.549771 7f556e3a8700 0 mds.0.server missing 1000002dd5c #1/iogen_45263.io (mine), will load later
2013-03-13 08:11:01.549773 7f556e3a8700 0 mds.0.server missing 10000033903 #1/iogen_54505.io (mine), will load later
2013-03-13 08:11:01.549777 7f556e3a8700 0 mds.0.server missing 1000003936f #1/iogen_60146.io (mine), will load later
2013-03-13 08:11:01.549780 7f556e3a8700 0 mds.0.server missing 1000003ee29 #1/iogen_4078.io (mine), will load later
2013-03-13 08:11:01.549782 7f556e3a8700 0 mds.0.server missing 10000044909 #1/iogen_28719.io (mine), will load later
2013-03-13 08:11:01.549785 7f556e3a8700 0 mds.0.server missing 1000004a4ce #1/iogen_37863.io (mine), will load later
2013-03-13 08:11:01.549788 7f556e3a8700 0 mds.0.server missing 1000005010c #1/iogen_46289.io (mine), will load later
2013-03-13 08:11:01.549791 7f556e3a8700 0 mds.0.server missing 10000055cff #1/iogen_54600.io (mine), will load later
2013-03-13 08:11:01.549793 7f556e3a8700 0 mds.0.server missing 1000005b838 #1/iogen_62979.io (mine), will load later
2013-03-13 08:11:01.549796 7f556e3a8700 0 mds.0.server missing 100000612cf #1/iogen_14616.io (mine), will load later
2013-03-13 08:11:01.549798 7f556e3a8700 0 mds.0.server missing 10000066e6f #1/iogen_43467.io (mine), will load later
2013-03-13 08:11:01.549802 7f556e3a8700 1 mds.0.19 reconnect_done
2013-03-13 08:11:02.195967 7f556e3a8700 1 mds.0.19 handle_mds_map i am now mds.0.19
2013-03-13 08:11:02.195971 7f556e3a8700 1 mds.0.19 handle_mds_map state change up:reconnect --> up:rejoin
2013-03-13 08:11:02.195974 7f556e3a8700 1 mds.0.19 rejoin_joint_start
2013-03-13 08:11:02.344789 7f556e3a8700 1 mds.0.19 rejoin_done
2013-03-13 08:11:04.323641 7f556e3a8700 1 mds.0.19 handle_mds_map i am now mds.0.19
2013-03-13 08:11:04.323645 7f556e3a8700 1 mds.0.19 handle_mds_map state change up:rejoin --> up:active
2013-03-13 08:11:04.323647 7f556e3a8700 1 mds.0.19 recovery_done -- successful recovery!
2013-03-13 08:11:04.324033 7f556e3a8700 1 mds.0.19 active_start
2013-03-13 08:11:04.327659 7f556e3a8700 1 mds.0.19 cluster recovered.
2013-03-13 08:32:19.978769 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6805/55437
2013-03-13 08:32:19.979221 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6805/55437
2013-03-13 08:32:22.784284 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6802/55225
2013-03-13 08:32:22.784873 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6802/55225
2013-03-13 08:47:20.030375 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6805/55437
2013-03-13 08:47:20.030821 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6805/55437
2013-03-13 08:47:22.848536 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6802/55225
2013-03-13 08:47:22.848897 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6802/55225
2013-03-13 09:02:20.069027 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6805/55437
2013-03-13 09:02:20.069480 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6805/55437
2013-03-13 09:02:22.871619 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6802/55225
2013-03-13 09:02:22.871969 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6802/55225
2013-03-13 09:17:20.127574 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6805/55437
2013-03-13 09:17:20.128020 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6805/55437
2013-03-13 09:17:22.972117 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6802/55225
2013-03-13 09:17:22.972493 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6802/55225
2013-03-13 09:32:20.183315 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6805/55437
2013-03-13 09:32:20.183723 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6805/55437
2013-03-13 09:32:23.016956 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6802/55225
2013-03-13 09:32:23.017384 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6802/55225
2013-03-13 09:47:20.184188 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6805/55437
2013-03-13 09:47:20.184590 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6805/55437
2013-03-13 09:47:23.040651 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6802/55225
2013-03-13 09:47:23.041015 7f556e3a8700 0 mds.0.19 ms_handle_connect on 10.214.136.10:6802/55225
2013-03-13 09:57:11.080632 7f556e3a8700 0 monclient: hunting for new mon
2013-03-13 09:57:11.080698 7f556e3a8700 0 mds.0.19 ms_handle_reset on 10.214.136.10:6789/0
2013-03-13 09:57:11.081006 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=16 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:20.478121 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:23.478380 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:26.478651 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:29.478958 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:32.479233 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:35.479499 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:38.479770 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:41.480044 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:42.699765 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:42.699854 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:42.783063 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:44.480288 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:47.480569 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:47.783177 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:50.480833 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:52.783325 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:53.481077 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:56.481355 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:57.783451 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:57:59.481615 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:02.481903 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:02.702126 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:02.702203 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:02.783537 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:05.482166 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:07.783692 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:08.482413 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:11.482699 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:12.783814 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:14.482930 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:17.483211 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:17.783893 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:20.483494 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:22.704251 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:22.704334 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:22.783958 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:23.483748 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:26.484031 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:27.784122 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:29.484270 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:32.484556 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:32.784214 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:35.484804 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:37.784343 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:38.485045 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:41.485325 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:42.706466 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:42.706544 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:42.784415 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:44.485598 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:47.485888 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:47.784525 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:50.486157 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:52.784680 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:53.486406 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:56.486705 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:57.784785 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:58:59.486985 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:02.487241 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:02.708682 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:02.708760 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:02.784856 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:05.487534 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:07.785016 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140000 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:08.487762 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0xa140780 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:11.488065 7f556c2a3700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230500 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:12.785131 7f556c3a4700 0 -- 10.214.136.10:6807/55002 >> 10.214.136.10:6789/0 pipe(0x3230a00 sd=20 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:14.350357 7f556eba9700 -1 mds.0.19 *** got signal Terminated ***
2013-03-13 09:59:14.350381 7f556eba9700 1 mds.0.19 suicide. wanted down:dne, now up:active
2013-03-13 09:59:14.351097 7f55738fb780 0 stopped.
2013-03-13 09:59:15.516549 7fdadf953700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
2013-03-13 09:59:15.923170 7fdadf953700 1 mds.0.20 handle_mds_map i am now mds.0.20
2013-03-13 09:59:15.923172 7fdadf953700 1 mds.0.20 handle_mds_map state change up:boot --> up:replay
2013-03-13 09:59:15.923174 7fdadf953700 1 mds.0.20 replay_start
2013-03-13 09:59:15.923178 7fdadf953700 1 mds.0.20 recovery set is
2013-03-13 09:59:15.923180 7fdadf953700 1 mds.0.20 need osdmap epoch 550, have 550
2013-03-13 09:59:15.923343 7fdadd04d700 0 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 09:59:15.923819 7fdadf953700 0 mds.0.20 ms_handle_connect on 10.214.136.10:6805/55437
2013-03-13 09:59:17.422046 7fdadf953700 0 mds.0.20 ms_handle_reset on 10.214.136.10:6805/55437
2013-03-13 09:59:17.422224 7fdadcd4a700 0 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 10:00:05.520600 7fdadf953700 0 mds.0.20 ms_handle_connect on 10.214.136.10:6801/64223
2013-03-13 10:00:19.818030 7fdadf953700 -1 *** Caught signal (Aborted) **
in thread 7fdadf953700

ceph version 0.58-500-gaf3b163 (af3b16349a49a8aee401e27c1b71fd704b31297c)
1: /usr/bin/ceph-mds() [0x84fa7a]
2: (()+0xfcb0) [0x7fdae3cb7cb0]
3: (gsignal()+0x35) [0x7fdae23a8425]
4: (abort()+0x17b) [0x7fdae23abb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fdae2cfa69d]
6: (()+0xb5846) [0x7fdae2cf8846]
7: (()+0xb5873) [0x7fdae2cf8873]
8: (()+0xb596e) [0x7fdae2cf896e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7adcd7]
10: (void decode<unsigned long, inodeno_t>(std::map<unsigned long, inodeno_t, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xde) [0x6b2cae]
11: (session_info_t::decode(ceph::buffer::list::iterator&)+0x139) [0x7d7ee9]
12: (SessionMap::decode(ceph::buffer::list::iterator&)+0xb98) [0x6c87e8]
13: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x83) [0x6c8cc3]
14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe38) [0x6f17b8]
15: (MDS::handle_core_message(Message*)+0xae8) [0x4db088]
16: (MDS::_dispatch(Message*)+0x2f) [0x4db24f]
17: (MDS::ms_dispatch(Message*)+0x1db) [0x4dcceb]
18: (DispatchQueue::entry()+0x341) [0x81e1a1]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x79c45d]
20: (()+0x7e9a) [0x7fdae3cafe9a]
21: (clone()+0x6d) [0x7fdae2465cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-258> 2013-03-13 09:59:15.513263 7fdae40d4780 5 asok(0x2aca000) register_command perfcounters_dump hook 0x2abf010
-257> 2013-03-13 09:59:15.513286 7fdae40d4780 5 asok(0x2aca000) register_command 1 hook 0x2abf010
-256> 2013-03-13 09:59:15.513291 7fdae40d4780 5 asok(0x2aca000) register_command perf dump hook 0x2abf010
-255> 2013-03-13 09:59:15.513300 7fdae40d4780 5 asok(0x2aca000) register_command perfcounters_schema hook 0x2abf010
-254> 2013-03-13 09:59:15.513304 7fdae40d4780 5 asok(0x2aca000) register_command 2 hook 0x2abf010
-253> 2013-03-13 09:59:15.513308 7fdae40d4780 5 asok(0x2aca000) register_command perf schema hook 0x2abf010
-252> 2013-03-13 09:59:15.513310 7fdae40d4780 5 asok(0x2aca000) register_command config show hook 0x2abf010
-251> 2013-03-13 09:59:15.513314 7fdae40d4780 5 asok(0x2aca000) register_command config set hook 0x2abf010
-250> 2013-03-13 09:59:15.513318 7fdae40d4780 5 asok(0x2aca000) register_command log flush hook 0x2abf010
-249> 2013-03-13 09:59:15.513320 7fdae40d4780 5 asok(0x2aca000) register_command log dump hook 0x2abf010
-248> 2013-03-13 09:59:15.513323 7fdae40d4780 5 asok(0x2aca000) register_command log reopen hook 0x2abf010
-247> 2013-03-13 09:59:15.514588 7fdae40d4780 0 ceph version 0.58-500-gaf3b163 (af3b16349a49a8aee401e27c1b71fd704b31297c), process ceph-mds, pid 63948
-246> 2013-03-13 09:59:15.514986 7fdae40d4780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/63948 need_addr=1
-245> 2013-03-13 09:59:15.515417 7fdae40d4780 1 finished global_init_daemonize
-244> 2013-03-13 09:59:15.515571 7fdae40d4780 5 asok(0x2aca000) init /var/run/ceph/ceph-mds.a.asok
-243> 2013-03-13 09:59:15.515587 7fdae40d4780 5 asok(0x2aca000) bind_and_listen /var/run/ceph/ceph-mds.a.asok
-242> 2013-03-13 09:59:15.515610 7fdae40d4780 5 asok(0x2aca000) register_command 0 hook 0x2abe038
-241> 2013-03-13 09:59:15.515617 7fdae40d4780 5 asok(0x2aca000) register_command version hook 0x2abe038
-240> 2013-03-13 09:59:15.515623 7fdae40d4780 5 asok(0x2aca000) register_command git_version hook 0x2abe038
-239> 2013-03-13 09:59:15.515628 7fdae40d4780 5 asok(0x2aca000) register_command help hook 0x2abf050
-238> 2013-03-13 09:59:15.515675 7fdae40d4780 10 monclient(hunting): build_initial_monmap
-237> 2013-03-13 09:59:15.515664 7fdae1156700 5 asok(0x2aca000) entry start
-236> 2013-03-13 09:59:15.515725 7fdae40d4780 1 -- 0.0.0.0:6800/63948 messenger.start
-235> 2013-03-13 09:59:15.515830 7fdae40d4780 5 adding auth protocol: cephx
-234> 2013-03-13 09:59:15.515837 7fdae40d4780 5 adding auth protocol: cephx
-233> 2013-03-13 09:59:15.515918 7fdae40d4780 1 accepter.accepter.start
-232> 2013-03-13 09:59:15.515942 7fdae40d4780 10 monclient(hunting): init
-231> 2013-03-13 09:59:15.515953 7fdae40d4780 5 adding auth protocol: cephx
-230> 2013-03-13 09:59:15.515955 7fdae40d4780 10 monclient(hunting): auth_supported 2 method cephx
-229> 2013-03-13 09:59:15.516148 7fdae40d4780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-a/keyring
-228> 2013-03-13 09:59:15.516212 7fdae40d4780 10 monclient(hunting): _reopen_session
-227> 2013-03-13 09:59:15.516259 7fdae40d4780 10 monclient(hunting): _pick_new_mon picked mon.a con 0x2ae0580 addr 10.214.136.10:6789/0
-226> 2013-03-13 09:59:15.516278 7fdae40d4780 10 monclient(hunting): _send_mon_message to mon.a at 10.214.136.10:6789/0
-225> 2013-03-13 09:59:15.516285 7fdae40d4780 1 -- 0.0.0.0:6800/63948 --> 10.214.136.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x2ad0400 con 0x2ae0580
-224> 2013-03-13 09:59:15.516297 7fdae40d4780 10 monclient(hunting): renew_subs
-223> 2013-03-13 09:59:15.516451 7fdae40d0700 1 -- 10.214.136.10:6800/63948 learned my addr 10.214.136.10:6800/63948
-222> 2013-03-13 09:59:15.516549 7fdadf953700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
-221> 2013-03-13 09:59:15.516836 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 1 ==== mon_map v1 ==== 191+0+0 (4269767265 0 0) 0x2ad0600 con 0x2ae0580
-220> 2013-03-13 09:59:15.516856 7fdadf953700 10 monclient(hunting): handle_monmap mon_map v1
-219> 2013-03-13 09:59:15.516865 7fdadf953700 10 monclient(hunting): got monmap 1, mon.a is now rank 0
-218> 2013-03-13 09:59:15.516868 7fdadf953700 10 monclient(hunting): dump:
epoch 1
fsid 3b3c3298-665f-4146-bdbf-376298b14e91
last_changed 2013-02-15 08:55:38.400848
created 2013-02-15 08:55:38.400848
0: 10.214.136.10:6789/0 mon.a

-217> 2013-03-13 09:59:15.516897 7fdadf953700 1 monclient(hunting): found mon.a
-216> 2013-03-13 09:59:15.516903 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3871252627 0 0) 0x2ad0a00 con 0x2ae0580
-215> 2013-03-13 09:59:15.516923 7fdadf953700 10 monclient: my global_id is 8598
-214> 2013-03-13 09:59:15.517121 7fdadf953700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-213> 2013-03-13 09:59:15.517127 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x2ad0600 con 0x2ae0580
-212> 2013-03-13 09:59:15.517447 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (1135520560 0 0) 0x2ad0800 con 0x2ae0580
-211> 2013-03-13 09:59:15.517523 7fdadf953700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-210> 2013-03-13 09:59:15.517527 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2ad0a00 con 0x2ae0580
-209> 2013-03-13 09:59:15.517870 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (2767124032 0 0) 0x2ad0c00 con 0x2ae0580
-208> 2013-03-13 09:59:15.517907 7fdadf953700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-207> 2013-03-13 09:59:15.517911 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x2ae4700 con 0x2ae0580
-206> 2013-03-13 09:59:15.517931 7fdadf953700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-03-13 09:58:45.517931)
-205> 2013-03-13 09:59:15.517940 7fdadf953700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-204> 2013-03-13 09:59:15.517943 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2ad0800 con 0x2ae0580
-203> 2013-03-13 09:59:15.517958 7fdae40d4780 5 monclient: authenticate success, global_id 8598
-202> 2013-03-13 09:59:15.517970 7fdae40d4780 10 monclient: wait_auth_rotating waiting (until 2013-03-13 09:59:45.517968)
-201> 2013-03-13 09:59:15.518061 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 5 ==== mon_map v1 ==== 191+0+0 (4269767265 0 0) 0x2ad0600 con 0x2ae0580
-200> 2013-03-13 09:59:15.518070 7fdadf953700 10 monclient: handle_monmap mon_map v1
-199> 2013-03-13 09:59:15.518074 7fdadf953700 10 monclient: got monmap 1, mon.a is now rank 0
-198> 2013-03-13 09:59:15.518075 7fdadf953700 10 monclient: dump:
epoch 1
fsid 3b3c3298-665f-4146-bdbf-376298b14e91
last_changed 2013-02-15 08:55:38.400848
created 2013-02-15 08:55:38.400848
0: 10.214.136.10:6789/0 mon.a

-197> 2013-03-13 09:59:15.518088 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2ae4a80 con 0x2ae0580
-196> 2013-03-13 09:59:15.518095 7fdadf953700 10 monclient: handle_subscribe_ack sent 2013-03-13 09:59:15.516300 renew after 2013-03-13 10:01:45.516300
-195> 2013-03-13 09:59:15.518184 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (4253469613 0 0) 0x2ad0400 con 0x2ae0580
-194> 2013-03-13 09:59:15.518235 7fdadf953700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 09:58:45.518234)
-193> 2013-03-13 09:59:15.518250 7fdae40d4780 10 monclient: wait_auth_rotating done
-192> 2013-03-13 09:59:15.518268 7fdae40d4780 5 asok(0x2aca000) register_command objecter_requests hook 0x2abf0d0
-191> 2013-03-13 09:59:15.518311 7fdae40d4780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-190> 2013-03-13 09:59:15.518320 7fdae40d4780 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:boot seq 1 v0) v2 -- ?+0 0x2aed000 con 0x2ae0580
-189> 2013-03-13 09:59:15.518339 7fdae40d4780 10 monclient: renew_subs
-188> 2013-03-13 09:59:15.518343 7fdae40d4780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-187> 2013-03-13 09:59:15.518347 7fdae40d4780 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x2ae4540 con 0x2ae0580
-186> 2013-03-13 09:59:15.518356 7fdae40d4780 10 monclient: renew_subs
-185> 2013-03-13 09:59:15.518359 7fdae40d4780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-184> 2013-03-13 09:59:15.518363 7fdae40d4780 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x2ae4e00 con 0x2ae0580
-183> 2013-03-13 09:59:15.922857 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 8 ==== osd_map(550..550 src has 49..550) v3 ==== 3054+0+0 (2203131030 0 0) 0x2af0600 con 0x2ae0580
-182> 2013-03-13 09:59:15.923003 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2ae4540 con 0x2ae0580
-181> 2013-03-13 09:59:15.923014 7fdadf953700 10 monclient: handle_subscribe_ack sent 2013-03-13 09:59:15.518341 renew after 2013-03-13 10:01:45.518341
-180> 2013-03-13 09:59:15.923121 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 10 ==== mdsmap(e 968) v1 ==== 583+0+0 (1663591001 0 0) 0x2af0400 con 0x2ae0580
-179> 2013-03-13 09:59:15.923136 7fdadf953700 5 mds.-1.0 handle_mds_map epoch 968 from mon.0
-178> 2013-03-13 09:59:15.923170 7fdadf953700 1 mds.0.20 handle_mds_map i am now mds.0.20
-177> 2013-03-13 09:59:15.923172 7fdadf953700 1 mds.0.20 handle_mds_map state change up:boot --> up:replay
-176> 2013-03-13 09:59:15.923174 7fdadf953700 1 mds.0.20 replay_start
-175> 2013-03-13 09:59:15.923178 7fdadf953700 1 mds.0.20 recovery set is
-174> 2013-03-13 09:59:15.923180 7fdadf953700 1 mds.0.20 need osdmap epoch 550, have 550
-173> 2013-03-13 09:59:15.923183 7fdadf953700 2 mds.0.20 boot_start 1: opening inotable
-172> 2013-03-13 09:59:15.923272 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- osd_op(mds.0.20:1 mds0_inotable [read 0~0] 1.b852b893 e550) v4 -- ?+0 0x2ac9480 con 0x2ae0840
-171> 2013-03-13 09:59:15.923291 7fdadf953700 2 mds.0.20 boot_start 1: opening sessionmap
-170> 2013-03-13 09:59:15.923302 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- osd_op(mds.0.20:2 mds0_sessionmap [read 0~0] 1.3270c60b e550) v4 -- ?+0 0x2ac9900 con 0x2ae0840
-169> 2013-03-13 09:59:15.923308 7fdadf953700 2 mds.0.20 boot_start 1: opening anchor table
-168> 2013-03-13 09:59:15.923291 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-167> 2013-03-13 09:59:15.923326 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- osd_op(mds.0.20:3 mds_anchortable [read 0~0] 1.a977f6a7 e550) v4 -- ?+0 0x2ac96c0 con 0x2ae0840
-166> 2013-03-13 09:59:15.923333 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-165> 2013-03-13 09:59:15.923343 7fdadd04d700 0 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
-164> 2013-03-13 09:59:15.923361 7fdadf953700 2 mds.0.20 boot_start 1: opening snap table
-163> 2013-03-13 09:59:15.923367 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-162> 2013-03-13 09:59:15.923376 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-161> 2013-03-13 09:59:15.923440 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6805/55437 -- osd_op(mds.0.20:4 mds_snaptable [read 0~0] 1.d90270ad e550) v4 -- ?+0 0x2ac9d80 con 0x2ae0b00
-160> 2013-03-13 09:59:15.923457 7fdadf953700 2 mds.0.20 boot_start 1: opening mds log
-159> 2013-03-13 09:59:15.923464 7fdadf953700 5 mds.0.log open discovering log bounds
-158> 2013-03-13 09:59:15.923470 7fdadf953700 1 mds.0.journaler(ro) recover start
-157> 2013-03-13 09:59:15.923474 7fdadf953700 1 mds.0.journaler(ro) read_head
-156> 2013-03-13 09:59:15.923488 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- osd_op(mds.0.20:5 200.00000000 [read 0~0] 1.844f3494 e550) v4 -- ?+0 0x2ac9b40 con 0x2ae0840
-155> 2013-03-13 09:59:15.923511 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 11 ==== osd_map(550..550 src has 49..550) v3 ==== 3054+0+0 (2203131030 0 0) 0x2af0200 con 0x2ae0580
-154> 2013-03-13 09:59:15.923532 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2ae4700 con 0x2ae0580
-153> 2013-03-13 09:59:15.923540 7fdadf953700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
-152> 2013-03-13 09:59:15.923547 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-151> 2013-03-13 09:59:15.923575 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-150> 2013-03-13 09:59:15.923819 7fdadf953700 0 mds.0.20 ms_handle_connect on 10.214.136.10:6805/55437
-149> 2013-03-13 09:59:16.138581 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== osd.1 10.214.136.10:6805/55437 1 ==== osd_op_reply(4 mds_snaptable [read 0~41] ondisk = 0) v4 ==== 112+0+41 (1682655365 0 2312406677) 0x2ad0400 con 0x2ae0b00
-148> 2013-03-13 09:59:16.323744 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-147> 2013-03-13 09:59:16.323775 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-146> 2013-03-13 09:59:17.123898 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-145> 2013-03-13 09:59:17.123935 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-144> 2013-03-13 09:59:17.421843 7fdadce4b700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2af5280 sd=18 :35536 s=2 pgs=219 cs=1 l=1).reader couldn't read tag, Success
-143> 2013-03-13 09:59:17.421914 7fdadce4b700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2af5280 sd=18 :35536 s=2 pgs=219 cs=1 l=1).fault 0: Success
-142> 2013-03-13 09:59:17.422046 7fdadf953700 0 mds.0.20 ms_handle_reset on 10.214.136.10:6805/55437
-141> 2013-03-13 09:59:17.422054 7fdadf953700 1 mds.0.objecter ms_handle_reset on osd.1
-140> 2013-03-13 09:59:17.422058 7fdadf953700 1 -- 10.214.136.10:6800/63948 mark_down 0x2ae0b00 -- pipe dne
-139> 2013-03-13 09:59:17.422128 7fdadf953700 10 monclient: renew_subs
-138> 2013-03-13 09:59:17.422134 7fdadf953700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-137> 2013-03-13 09:59:17.422138 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=969+,monmap=2+,osdmap=551}) v2 -- ?+0 0x2ae4700 con 0x2ae0580
-136> 2013-03-13 09:59:17.422167 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=19 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-135> 2013-03-13 09:59:17.422214 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-134> 2013-03-13 09:59:17.422224 7fdadcd4a700 0 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault
-133> 2013-03-13 09:59:17.422259 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-132> 2013-03-13 09:59:17.422268 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-131> 2013-03-13 09:59:17.422413 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 13 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2ae4540 con 0x2ae0580
-130> 2013-03-13 09:59:17.422428 7fdadf953700 10 monclient: handle_subscribe_ack sent 2013-03-13 09:59:17.422131 renew after 2013-03-13 10:01:47.422131
-129> 2013-03-13 09:59:17.622430 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-128> 2013-03-13 09:59:17.622468 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-127> 2013-03-13 09:59:18.022592 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-126> 2013-03-13 09:59:18.022634 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-125> 2013-03-13 09:59:18.516321 7fdade951700 10 monclient: tick
-124> 2013-03-13 09:59:18.516347 7fdade951700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 09:58:48.516345)
-123> 2013-03-13 09:59:18.516359 7fdade951700 10 monclient: renew subs? (now: 2013-03-13 09:59:18.516359; renew after: 2013-03-13 10:01:47.422131) -- no
-122> 2013-03-13 09:59:18.724099 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-121> 2013-03-13 09:59:18.724129 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-120> 2013-03-13 09:59:18.822808 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-119> 2013-03-13 09:59:18.822841 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-118> 2013-03-13 09:59:19.518470 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-117> 2013-03-13 09:59:19.518503 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 2 v968) v2 -- ?+0 0x2aed2c0 con 0x2ae0580
-116> 2013-03-13 09:59:19.518868 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 14 ==== mdsbeacon(8598/a up:replay seq 2 v968) v2 ==== 103+0+0 (627647807 0 0) 0x2aed000 con 0x2ae0580
-115> 2013-03-13 09:59:20.423004 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-114> 2013-03-13 09:59:20.423042 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-113> 2013-03-13 09:59:21.924305 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-112> 2013-03-13 09:59:21.924343 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-111> 2013-03-13 09:59:23.518665 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-110> 2013-03-13 09:59:23.518688 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 3 v968) v2 -- ?+0 0x2aed580 con 0x2ae0580
-109> 2013-03-13 09:59:23.519031 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 15 ==== mdsbeacon(8598/a up:replay seq 3 v968) v2 ==== 103+0+0 (1547720792 0 0) 0x2aedb00 con 0x2ae0580
-108> 2013-03-13 09:59:23.623223 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-107> 2013-03-13 09:59:23.623262 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-106> 2013-03-13 09:59:27.518852 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-105> 2013-03-13 09:59:27.518875 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 4 v968) v2 -- ?+0 0x2aed2c0 con 0x2ae0580
-104> 2013-03-13 09:59:27.519267 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 16 ==== mdsbeacon(8598/a up:replay seq 4 v968) v2 ==== 103+0+0 (913464476 0 0) 0x2aed840 con 0x2ae0580
-103> 2013-03-13 09:59:28.324517 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-102> 2013-03-13 09:59:28.324555 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-101> 2013-03-13 09:59:28.516501 7fdade951700 10 monclient: tick
-100> 2013-03-13 09:59:28.516524 7fdade951700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 09:58:58.516522)
-99> 2013-03-13 09:59:28.516538 7fdade951700 10 monclient: renew subs? (now: 2013-03-13 09:59:28.516537; renew after: 2013-03-13 10:01:47.422131) -- no
-98> 2013-03-13 09:59:30.023459 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-97> 2013-03-13 09:59:30.023528 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-96> 2013-03-13 09:59:30.518716 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- ping v1 -- ?+0 0x2b02000 con 0x2ae0840
-95> 2013-03-13 09:59:30.518821 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-94> 2013-03-13 09:59:30.518857 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-93> 2013-03-13 09:59:31.519025 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-92> 2013-03-13 09:59:31.519048 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 5 v968) v2 -- ?+0 0x2aeddc0 con 0x2ae0580
-91> 2013-03-13 09:59:31.519412 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 17 ==== mdsbeacon(8598/a up:replay seq 5 v968) v2 ==== 103+0+0 (1331372539 0 0) 0x2aeddc0 con 0x2ae0580
-90> 2013-03-13 09:59:35.518824 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- ping v1 -- ?+0 0x2b02380 con 0x2ae0840
-89> 2013-03-13 09:59:35.518930 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-88> 2013-03-13 09:59:35.518966 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-87> 2013-03-13 09:59:35.519126 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-86> 2013-03-13 09:59:35.519132 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 6 v968) v2 -- ?+0 0x2aedb00 con 0x2ae0580
-85> 2013-03-13 09:59:35.519484 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 18 ==== mdsbeacon(8598/a up:replay seq 6 v968) v2 ==== 103+0+0 (3290474066 0 0) 0x2aed2c0 con 0x2ae0580
-84> 2013-03-13 09:59:38.516674 7fdade951700 10 monclient: tick
-83> 2013-03-13 09:59:38.516698 7fdade951700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 09:59:08.516696)
-82> 2013-03-13 09:59:38.516710 7fdade951700 10 monclient: renew subs? (now: 2013-03-13 09:59:38.516710; renew after: 2013-03-13 10:01:47.422131) -- no
-81> 2013-03-13 09:59:39.519276 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-80> 2013-03-13 09:59:39.519298 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 7 v968) v2 -- ?+0 0x2aed000 con 0x2ae0580
-79> 2013-03-13 09:59:39.519589 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 19 ==== mdsbeacon(8598/a up:replay seq 7 v968) v2 ==== 103+0+0 (3171539765 0 0) 0x2aed580 con 0x2ae0580
-78> 2013-03-13 09:59:40.518983 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- ping v1 -- ?+0 0x2b021c0 con 0x2ae0840
-77> 2013-03-13 09:59:40.519065 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-76> 2013-03-13 09:59:40.519095 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-75> 2013-03-13 09:59:42.823706 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-74> 2013-03-13 09:59:42.823745 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-73> 2013-03-13 09:59:43.519403 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-72> 2013-03-13 09:59:43.519427 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 8 v968) v2 -- ?+0 0x2aee080 con 0x2ae0580
-71> 2013-03-13 09:59:43.519813 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 20 ==== mdsbeacon(8598/a up:replay seq 8 v968) v2 ==== 103+0+0 (272936922 0 0) 0x2aeeb80 con 0x2ae0580
-70> 2013-03-13 09:59:45.519096 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- ping v1 -- ?+0 0x2b028c0 con 0x2ae0840
-69> 2013-03-13 09:59:45.519208 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-68> 2013-03-13 09:59:45.519242 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-67> 2013-03-13 09:59:47.519599 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-66> 2013-03-13 09:59:47.519621 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 9 v968) v2 -- ?+0 0x2aed2c0 con 0x2ae0580
-65> 2013-03-13 09:59:47.520002 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 21 ==== mdsbeacon(8598/a up:replay seq 9 v968) v2 ==== 103+0+0 (1768805053 0 0) 0x2aee8c0 con 0x2ae0580
-64> 2013-03-13 09:59:48.516847 7fdade951700 10 monclient: tick
-63> 2013-03-13 09:59:48.516869 7fdade951700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 09:59:18.516867)
-62> 2013-03-13 09:59:48.516884 7fdade951700 10 monclient: renew subs? (now: 2013-03-13 09:59:48.516884; renew after: 2013-03-13 10:01:47.422131) -- no
-61> 2013-03-13 09:59:50.519264 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6802/55225 -- ping v1 -- ?+0 0x2b02700 con 0x2ae0840
-60> 2013-03-13 09:59:50.519366 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/55225, 111: Connection refused
-59> 2013-03-13 09:59:50.519402 7fdadd04d700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6802/55225 pipe(0x2af5500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-58> 2013-03-13 09:59:51.519732 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-57> 2013-03-13 09:59:51.519755 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 10 v968) v2 -- ?+0 0x2aeddc0 con 0x2ae0580
-56> 2013-03-13 09:59:51.520127 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 22 ==== mdsbeacon(8598/a up:replay seq 10 v968) v2 ==== 103+0+0 (3793114388 0 0) 0x2aee600 con 0x2ae0580
-55> 2013-03-13 09:59:53.479479 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 23 ==== osd_map(551..551 src has 50..551) v3 ==== 169+0+0 (3442193004 0 0) 0x2af0000 con 0x2ae0580
-54> 2013-03-13 09:59:53.479546 7fdadf953700 1 -- 10.214.136.10:6800/63948 mark_down 0x2ae0840 -- 0x2af5500
-53> 2013-03-13 09:59:53.479569 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6805/55437 -- osd_op(mds.0.20:1 mds0_inotable [read 0~0] 1.b852b893 RETRY=1 e551) v4 -- ?+0 0x2b066c0 con 0x2ae09a0
-52> 2013-03-13 09:59:53.479636 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6805/55437 -- osd_op(mds.0.20:2 mds0_sessionmap [read 0~0] 1.3270c60b RETRY=1 e551) v4 -- ?+0 0x2b06480 con 0x2ae09a0
-51> 2013-03-13 09:59:53.479666 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6805/55437 -- osd_op(mds.0.20:3 mds_anchortable [read 0~0] 1.a977f6a7 RETRY=1 e551) v4 -- ?+0 0x2b06240 con 0x2ae09a0
-50> 2013-03-13 09:59:53.479675 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6805/55437 -- osd_op(mds.0.20:5 200.00000000 [read 0~0] 1.844f3494 RETRY=1 e551) v4 -- ?+0 0x2b06000 con 0x2ae09a0
-49> 2013-03-13 09:59:53.479672 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-48> 2013-03-13 09:59:53.479704 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-47> 2013-03-13 09:59:55.519913 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-46> 2013-03-13 09:59:55.519931 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 11 v968) v2 -- ?+0 0x2aed840 con 0x2ae0580
-45> 2013-03-13 09:59:55.597145 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 24 ==== mdsbeacon(8598/a up:replay seq 11 v968) v2 ==== 103+0+0 (2604607603 0 0) 0x2aee340 con 0x2ae0580
-44> 2013-03-13 09:59:58.517019 7fdade951700 10 monclient: tick
-43> 2013-03-13 09:59:58.517042 7fdade951700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 09:59:28.517040)
-42> 2013-03-13 09:59:58.517055 7fdade951700 10 monclient: renew subs? (now: 2013-03-13 09:59:58.517055; renew after: 2013-03-13 10:01:47.422131) -- no
-41> 2013-03-13 09:59:59.520115 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-40> 2013-03-13 09:59:59.520141 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 12 v968) v2 -- ?+0 0x2aedb00 con 0x2ae0580
-39> 2013-03-13 09:59:59.955989 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 25 ==== mdsbeacon(8598/a up:replay seq 12 v968) v2 ==== 103+0+0 (4044172471 0 0) 0x2aed2c0 con 0x2ae0580
-38> 2013-03-13 10:00:03.520272 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-37> 2013-03-13 10:00:03.520295 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 13 v968) v2 -- ?+0 0x2aee600 con 0x2ae0580
-36> 2013-03-13 10:00:03.920918 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 26 ==== mdsbeacon(8598/a up:replay seq 13 v968) v2 ==== 103+0+0 (2284081616 0 0) 0x2aee080 con 0x2ae0580
-35> 2013-03-13 10:00:05.519602 7fdadd84e700 10 monclient: renew_subs
-34> 2013-03-13 10:00:05.519616 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-33> 2013-03-13 10:00:05.519637 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=969+,monmap=2+,osdmap=552}) v2 -- ?+0 0x2b02540 con 0x2ae0580
-32> 2013-03-13 10:00:05.519660 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6805/55437 -- ping v1 -- ?+0 0x2b02380 con 0x2ae09a0
-31> 2013-03-13 10:00:05.519756 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/55437, 111: Connection refused
-30> 2013-03-13 10:00:05.519799 7fdadcd4a700 2 -- 10.214.136.10:6800/63948 >> 10.214.136.10:6805/55437 pipe(0x2ae3c80 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-29> 2013-03-13 10:00:05.520052 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 27 ==== osd_map(552..555 src has 54..555) v3 ==== 1513+0+0 (951697260 0 0) 0x2af0c00 con 0x2ae0580
-28> 2013-03-13 10:00:05.520156 7fdadf953700 1 -- 10.214.136.10:6800/63948 mark_down 0x2ae09a0 -- 0x2ae3c80
-27> 2013-03-13 10:00:05.520192 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6801/64223 -- osd_op(mds.0.20:1 mds0_inotable [read 0~0] 1.b852b893 RETRY=2 e555) v4 -- ?+0 0x2ac9480 con 0x2ae0c60
-26> 2013-03-13 10:00:05.520292 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6801/64223 -- osd_op(mds.0.20:2 mds0_sessionmap [read 0~0] 1.3270c60b RETRY=2 e555) v4 -- ?+0 0x2ac9900 con 0x2ae0c60
-25> 2013-03-13 10:00:05.520305 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6801/64223 -- osd_op(mds.0.20:3 mds_anchortable [read 0~0] 1.a977f6a7 RETRY=2 e555) v4 -- ?+0 0x2ac96c0 con 0x2ae0c60
-24> 2013-03-13 10:00:05.520312 7fdadf953700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6801/64223 -- osd_op(mds.0.20:5 200.00000000 [read 0~0] 1.844f3494 RETRY=2 e555) v4 -- ?+0 0x2ac9b40 con 0x2ae0c60
-23> 2013-03-13 10:00:05.520331 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 28 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2ae4a80 con 0x2ae0580
-22> 2013-03-13 10:00:05.520338 7fdadf953700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:00:05.519613 renew after 2013-03-13 10:02:35.519613
-21> 2013-03-13 10:00:05.520600 7fdadf953700 0 mds.0.20 ms_handle_connect on 10.214.136.10:6801/64223
-20> 2013-03-13 10:00:07.520447 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-19> 2013-03-13 10:00:07.520470 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 14 v968) v2 -- ?+0 0x2aee8c0 con 0x2ae0580
-18> 2013-03-13 10:00:07.520733 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 29 ==== mdsbeacon(8598/a up:replay seq 14 v968) v2 ==== 103+0+0 (56612473 0 0) 0x2aed000 con 0x2ae0580
-17> 2013-03-13 10:00:08.517190 7fdade951700 10 monclient: tick
-16> 2013-03-13 10:00:08.517215 7fdade951700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 09:59:38.517213)
-15> 2013-03-13 10:00:08.517230 7fdade951700 10 monclient: renew subs? (now: 2013-03-13 10:00:08.517230; renew after: 2013-03-13 10:02:35.519613) -- no
-14> 2013-03-13 10:00:11.520590 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-13> 2013-03-13 10:00:11.520608 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 15 v968) v2 -- ?+0 0x2aeeb80 con 0x2ae0580
-12> 2013-03-13 10:00:11.520965 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 30 ==== mdsbeacon(8598/a up:replay seq 15 v968) v2 ==== 103+0+0 (2054597406 0 0) 0x2aee8c0 con 0x2ae0580
-11> 2013-03-13 10:00:12.879156 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== osd.0 10.214.136.10:6801/64223 1 ==== osd_op_reply(1 mds0_inotable [read 0~29] ondisk = 0) v4 ==== 112+0+29 (3404241379 0 2795693049) 0x2ad0400 con 0x2ae0c60
-10> 2013-03-13 10:00:15.520703 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-9> 2013-03-13 10:00:15.520722 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 16 v968) v2 -- ?+0 0x2aed580 con 0x2ae0580
-8> 2013-03-13 10:00:15.521043 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 31 ==== mdsbeacon(8598/a up:replay seq 16 v968) v2 ==== 103+0+0 (1546209622 0 0) 0x2aee600 con 0x2ae0580
-7> 2013-03-13 10:00:18.517363 7fdade951700 10 monclient: tick
-6> 2013-03-13 10:00:18.517386 7fdade951700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 09:59:48.517384)
-5> 2013-03-13 10:00:18.517401 7fdade951700 10 monclient: renew subs? (now: 2013-03-13 10:00:18.517401; renew after: 2013-03-13 10:02:35.519613) -- no
-4> 2013-03-13 10:00:19.520832 7fdadd84e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-3> 2013-03-13 10:00:19.520859 7fdadd84e700 1 -- 10.214.136.10:6800/63948 --> 10.214.136.10:6789/0 -- mdsbeacon(8598/a up:replay seq 17 v968) v2 -- ?+0 0x2aee8c0 con 0x2ae0580
-2> 2013-03-13 10:00:19.521235 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== mon.0 10.214.136.10:6789/0 32 ==== mdsbeacon(8598/a up:replay seq 17 v968) v2 ==== 103+0+0 (620770353 0 0) 0x2aedb00 con 0x2ae0580
-1> 2013-03-13 10:00:19.816360 7fdadf953700 1 -- 10.214.136.10:6800/63948 <== osd.0 10.214.136.10:6801/64223 2 ==== osd_op_reply(2 mds0_sessionmap [read 0~1848] ondisk = 0) v4 ==== 114+0+1848 (2189189684 0 1211354084) 0x2af0000 con 0x2ae0c60
0> 2013-03-13 10:00:19.818030 7fdadf953700 -1 *** Caught signal (Aborted) **
in thread 7fdadf953700

ceph version 0.58-500-gaf3b163 (af3b16349a49a8aee401e27c1b71fd704b31297c)
1: /usr/bin/ceph-mds() [0x84fa7a]
2: (()+0xfcb0) [0x7fdae3cb7cb0]
3: (gsignal()+0x35) [0x7fdae23a8425]
4: (abort()+0x17b) [0x7fdae23abb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fdae2cfa69d]
6: (()+0xb5846) [0x7fdae2cf8846]
7: (()+0xb5873) [0x7fdae2cf8873]
8: (()+0xb596e) [0x7fdae2cf896e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7adcd7]
10: (void decode<unsigned long, inodeno_t>(std::map<unsigned long, inodeno_t, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xde) [0x6b2cae]
11: (session_info_t::decode(ceph::buffer::list::iterator&)+0x139) [0x7d7ee9]
12: (SessionMap::decode(ceph::buffer::list::iterator&)+0xb98) [0x6c87e8]
13: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x83) [0x6c8cc3]
14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe38) [0x6f17b8]
15: (MDS::handle_core_message(Message*)+0xae8) [0x4db088]
16: (MDS::_dispatch(Message*)+0x2f) [0x4db24f]
17: (MDS::ms_dispatch(Message*)+0x1db) [0x4dcceb]
18: (DispatchQueue::entry()+0x341) [0x81e1a1]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x79c45d]
20: (()+0x7e9a) [0x7fdae3cafe9a]
21: (clone()+0x6d) [0x7fdae2465cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.a.log
--- end dump of recent events ---
2013-03-13 10:05:09.676144 7f9a6eca3700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
2013-03-13 10:05:10.357778 7f9a6eca3700 1 mds.0.21 handle_mds_map i am now mds.0.21
2013-03-13 10:05:10.357782 7f9a6eca3700 1 mds.0.21 handle_mds_map state change up:boot --> up:replay
2013-03-13 10:05:10.357784 7f9a6eca3700 1 mds.0.21 replay_start
2013-03-13 10:05:10.357788 7f9a6eca3700 1 mds.0.21 recovery set is
2013-03-13 10:05:10.357789 7f9a6eca3700 1 mds.0.21 need osdmap epoch 556, have 556
2013-03-13 10:05:10.357959 7f9a6c39d700 0 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 10:05:10.358356 7f9a6eca3700 0 mds.0.21 ms_handle_connect on 10.214.136.10:6806/64507
2013-03-13 10:05:11.384396 7f9a6eca3700 0 mds.0.21 ms_handle_reset on 10.214.136.10:6806/64507
2013-03-13 10:05:11.384626 7f9a6c19b700 0 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 10:05:14.859466 7f9a6c39d700 0 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :37065 s=1 pgs=0 cs=0 l=1).connect claims to be 0.0.0.0:6801/6015 not 10.214.136.10:6801/64223 - wrong node!
2013-03-13 10:05:29.941503 7f9a6eca3700 0 mds.0.21 ms_handle_connect on 10.214.136.10:6801/6015
2013-03-13 10:05:32.849786 7f9a6eca3700 -1 *** Caught signal (Aborted) **
in thread 7f9a6eca3700

ceph version 0.58-500-gaf3b163 (af3b16349a49a8aee401e27c1b71fd704b31297c)
1: /usr/bin/ceph-mds() [0x84fa7a]
2: (()+0xfcb0) [0x7f9a73007cb0]
3: (gsignal()+0x35) [0x7f9a716f8425]
4: (abort()+0x17b) [0x7f9a716fbb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9a7204a69d]
6: (()+0xb5846) [0x7f9a72048846]
7: (()+0xb5873) [0x7f9a72048873]
8: (()+0xb596e) [0x7f9a7204896e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7adcd7]
10: (void decode<unsigned long, inodeno_t>(std::map<unsigned long, inodeno_t, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xde) [0x6b2cae]
11: (session_info_t::decode(ceph::buffer::list::iterator&)+0x139) [0x7d7ee9]
12: (SessionMap::decode(ceph::buffer::list::iterator&)+0xb98) [0x6c87e8]
13: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x83) [0x6c8cc3]
14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe38) [0x6f17b8]
15: (MDS::handle_core_message(Message*)+0xae8) [0x4db088]
16: (MDS::_dispatch(Message*)+0x2f) [0x4db24f]
17: (MDS::ms_dispatch(Message*)+0x1db) [0x4dcceb]
18: (DispatchQueue::entry()+0x341) [0x81e1a1]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x79c45d]
20: (()+0x7e9a) [0x7f9a72fffe9a]
21: (clone()+0x6d) [0x7f9a717b5cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-188> 2013-03-13 10:05:09.672824 7f9a73424780 5 asok(0x2c93000) register_command perfcounters_dump hook 0x2c88010
-187> 2013-03-13 10:05:09.672847 7f9a73424780 5 asok(0x2c93000) register_command 1 hook 0x2c88010
-186> 2013-03-13 10:05:09.672849 7f9a73424780 5 asok(0x2c93000) register_command perf dump hook 0x2c88010
-185> 2013-03-13 10:05:09.672857 7f9a73424780 5 asok(0x2c93000) register_command perfcounters_schema hook 0x2c88010
-184> 2013-03-13 10:05:09.672861 7f9a73424780 5 asok(0x2c93000) register_command 2 hook 0x2c88010
-183> 2013-03-13 10:05:09.672863 7f9a73424780 5 asok(0x2c93000) register_command perf schema hook 0x2c88010
-182> 2013-03-13 10:05:09.672867 7f9a73424780 5 asok(0x2c93000) register_command config show hook 0x2c88010
-181> 2013-03-13 10:05:09.672870 7f9a73424780 5 asok(0x2c93000) register_command config set hook 0x2c88010
-180> 2013-03-13 10:05:09.672873 7f9a73424780 5 asok(0x2c93000) register_command log flush hook 0x2c88010
-179> 2013-03-13 10:05:09.672875 7f9a73424780 5 asok(0x2c93000) register_command log dump hook 0x2c88010
-178> 2013-03-13 10:05:09.672878 7f9a73424780 5 asok(0x2c93000) register_command log reopen hook 0x2c88010
-177> 2013-03-13 10:05:09.674170 7f9a73424780 0 ceph version 0.58-500-gaf3b163 (af3b16349a49a8aee401e27c1b71fd704b31297c), process ceph-mds, pid 5730
-176> 2013-03-13 10:05:09.674560 7f9a73424780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/5730 need_addr=1
-175> 2013-03-13 10:05:09.675016 7f9a73424780 1 finished global_init_daemonize
-174> 2013-03-13 10:05:09.675166 7f9a73424780 5 asok(0x2c93000) init /var/run/ceph/ceph-mds.a.asok
-173> 2013-03-13 10:05:09.675182 7f9a73424780 5 asok(0x2c93000) bind_and_listen /var/run/ceph/ceph-mds.a.asok
-172> 2013-03-13 10:05:09.675219 7f9a73424780 5 asok(0x2c93000) register_command 0 hook 0x2c87038
-171> 2013-03-13 10:05:09.675226 7f9a73424780 5 asok(0x2c93000) register_command version hook 0x2c87038
-170> 2013-03-13 10:05:09.675231 7f9a73424780 5 asok(0x2c93000) register_command git_version hook 0x2c87038
-169> 2013-03-13 10:05:09.675237 7f9a73424780 5 asok(0x2c93000) register_command help hook 0x2c88050
-168> 2013-03-13 10:05:09.675285 7f9a73424780 10 monclient(hunting): build_initial_monmap
-167> 2013-03-13 10:05:09.675276 7f9a704a6700 5 asok(0x2c93000) entry start
-166> 2013-03-13 10:05:09.675335 7f9a73424780 1 -- 0.0.0.0:6800/5730 messenger.start
-165> 2013-03-13 10:05:09.675453 7f9a73424780 5 adding auth protocol: cephx
-164> 2013-03-13 10:05:09.675460 7f9a73424780 5 adding auth protocol: cephx
-163> 2013-03-13 10:05:09.675540 7f9a73424780 1 accepter.accepter.start
-162> 2013-03-13 10:05:09.675564 7f9a73424780 10 monclient(hunting): init
-161> 2013-03-13 10:05:09.675570 7f9a73424780 5 adding auth protocol: cephx
-160> 2013-03-13 10:05:09.675571 7f9a73424780 10 monclient(hunting): auth_supported 2 method cephx
-159> 2013-03-13 10:05:09.675754 7f9a73424780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-a/keyring
-158> 2013-03-13 10:05:09.675817 7f9a73424780 10 monclient(hunting): _reopen_session
-157> 2013-03-13 10:05:09.675865 7f9a73424780 10 monclient(hunting): _pick_new_mon picked mon.a con 0x2ca9580 addr 10.214.136.10:6789/0
-156> 2013-03-13 10:05:09.675881 7f9a73424780 10 monclient(hunting): _send_mon_message to mon.a at 10.214.136.10:6789/0
-155> 2013-03-13 10:05:09.675887 7f9a73424780 1 -- 0.0.0.0:6800/5730 --> 10.214.136.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x2c99400 con 0x2ca9580
-154> 2013-03-13 10:05:09.675900 7f9a73424780 10 monclient(hunting): renew_subs
-153> 2013-03-13 10:05:09.676044 7f9a73420700 1 -- 10.214.136.10:6800/5730 learned my addr 10.214.136.10:6800/5730
-152> 2013-03-13 10:05:09.676144 7f9a6eca3700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
-151> 2013-03-13 10:05:09.856016 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 1 ==== mon_map v1 ==== 191+0+0 (4269767265 0 0) 0x2c99600 con 0x2ca9580
-150> 2013-03-13 10:05:09.856063 7f9a6eca3700 10 monclient(hunting): handle_monmap mon_map v1
-149> 2013-03-13 10:05:09.856073 7f9a6eca3700 10 monclient(hunting): got monmap 1, mon.a is now rank 0
-148> 2013-03-13 10:05:09.856076 7f9a6eca3700 10 monclient(hunting): dump:
epoch 1
fsid 3b3c3298-665f-4146-bdbf-376298b14e91
last_changed 2013-02-15 08:55:38.400848
created 2013-02-15 08:55:38.400848
0: 10.214.136.10:6789/0 mon.a

-147> 2013-03-13 10:05:09.856116 7f9a6eca3700 1 monclient(hunting): found mon.a
-146> 2013-03-13 10:05:09.856124 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (1818530492 0 0) 0x2c99a00 con 0x2ca9580
-145> 2013-03-13 10:05:09.856149 7f9a6eca3700 10 monclient: my global_id is 8999
-144> 2013-03-13 10:05:09.856370 7f9a6eca3700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-143> 2013-03-13 10:05:09.856376 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x2c99600 con 0x2ca9580
-142> 2013-03-13 10:05:09.856693 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (197284069 0 0) 0x2c99800 con 0x2ca9580
-141> 2013-03-13 10:05:09.856794 7f9a6eca3700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-140> 2013-03-13 10:05:09.856800 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2c99a00 con 0x2ca9580
-139> 2013-03-13 10:05:09.857184 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (3328704297 0 0) 0x2c99c00 con 0x2ca9580
-138> 2013-03-13 10:05:09.857243 7f9a6eca3700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-137> 2013-03-13 10:05:09.857248 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x2cad700 con 0x2ca9580
-136> 2013-03-13 10:05:09.857283 7f9a6eca3700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-03-13 10:04:39.857282)
-135> 2013-03-13 10:05:09.857294 7f9a6eca3700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-134> 2013-03-13 10:05:09.857296 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2c99800 con 0x2ca9580
-133> 2013-03-13 10:05:09.857329 7f9a73424780 5 monclient: authenticate success, global_id 8999
-132> 2013-03-13 10:05:09.857335 7f9a73424780 10 monclient: wait_auth_rotating waiting (until 2013-03-13 10:05:39.857335)
-131> 2013-03-13 10:05:09.857433 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 5 ==== mon_map v1 ==== 191+0+0 (4269767265 0 0) 0x2c99600 con 0x2ca9580
-130> 2013-03-13 10:05:09.857451 7f9a6eca3700 10 monclient: handle_monmap mon_map v1
-129> 2013-03-13 10:05:09.857459 7f9a6eca3700 10 monclient: got monmap 1, mon.a is now rank 0
-128> 2013-03-13 10:05:09.857461 7f9a6eca3700 10 monclient: dump:
epoch 1
fsid 3b3c3298-665f-4146-bdbf-376298b14e91
last_changed 2013-02-15 08:55:38.400848
created 2013-02-15 08:55:38.400848
0: 10.214.136.10:6789/0 mon.a

-127> 2013-03-13 10:05:09.857477 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2cada80 con 0x2ca9580
-126> 2013-03-13 10:05:09.857485 7f9a6eca3700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:05:09.675903 renew after 2013-03-13 10:07:39.675903
-125> 2013-03-13 10:05:09.857566 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (437283359 0 0) 0x2c99400 con 0x2ca9580
-124> 2013-03-13 10:05:09.857613 7f9a6eca3700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 10:04:39.857612)
-123> 2013-03-13 10:05:09.857630 7f9a73424780 10 monclient: wait_auth_rotating done
-122> 2013-03-13 10:05:09.857650 7f9a73424780 5 asok(0x2c93000) register_command objecter_requests hook 0x2c880d0
-121> 2013-03-13 10:05:09.857704 7f9a73424780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-120> 2013-03-13 10:05:09.857713 7f9a73424780 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mdsbeacon(8999/a up:boot seq 1 v0) v2 -- ?+0 0x2cb6000 con 0x2ca9580
-119> 2013-03-13 10:05:09.857732 7f9a73424780 10 monclient: renew_subs
-118> 2013-03-13 10:05:09.857737 7f9a73424780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-117> 2013-03-13 10:05:09.857741 7f9a73424780 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x2cad540 con 0x2ca9580
-116> 2013-03-13 10:05:09.857764 7f9a73424780 10 monclient: renew_subs
-115> 2013-03-13 10:05:09.857775 7f9a73424780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-114> 2013-03-13 10:05:09.857784 7f9a73424780 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x2cade00 con 0x2ca9580
-113> 2013-03-13 10:05:10.357562 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 8 ==== osd_map(556..556 src has 55..556) v3 ==== 3198+0+0 (1670496509 0 0) 0x2cbe600 con 0x2ca9580
-112> 2013-03-13 10:05:10.357717 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2cbf1c0 con 0x2ca9580
-111> 2013-03-13 10:05:10.357726 7f9a6eca3700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:05:09.857735 renew after 2013-03-13 10:07:39.857735
-110> 2013-03-13 10:05:10.357737 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 10 ==== mdsmap(e 989) v1 ==== 583+0+0 (4111995737 0 0) 0x2cbe400 con 0x2ca9580
-109> 2013-03-13 10:05:10.357748 7f9a6eca3700 5 mds.-1.0 handle_mds_map epoch 989 from mon.0
-108> 2013-03-13 10:05:10.357778 7f9a6eca3700 1 mds.0.21 handle_mds_map i am now mds.0.21
-107> 2013-03-13 10:05:10.357782 7f9a6eca3700 1 mds.0.21 handle_mds_map state change up:boot --> up:replay
-106> 2013-03-13 10:05:10.357784 7f9a6eca3700 1 mds.0.21 replay_start
-105> 2013-03-13 10:05:10.357788 7f9a6eca3700 1 mds.0.21 recovery set is
-104> 2013-03-13 10:05:10.357789 7f9a6eca3700 1 mds.0.21 need osdmap epoch 556, have 556
-103> 2013-03-13 10:05:10.357793 7f9a6eca3700 2 mds.0.21 boot_start 1: opening inotable
-102> 2013-03-13 10:05:10.357878 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6801/64223 -- osd_op(mds.0.21:1 mds0_inotable [read 0~0] 1.b852b893 e556) v4 -- ?+0 0x2c92480 con 0x2ca9840
-101> 2013-03-13 10:05:10.357899 7f9a6eca3700 2 mds.0.21 boot_start 1: opening sessionmap
-100> 2013-03-13 10:05:10.357911 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6801/64223 -- osd_op(mds.0.21:2 mds0_sessionmap [read 0~0] 1.3270c60b e556) v4 -- ?+0 0x2c92900 con 0x2ca9840
-99> 2013-03-13 10:05:10.357919 7f9a6eca3700 2 mds.0.21 boot_start 1: opening anchor table
-98> 2013-03-13 10:05:10.357908 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/64223, 111: Connection refused
-97> 2013-03-13 10:05:10.357933 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6801/64223 -- osd_op(mds.0.21:3 mds_anchortable [read 0~0] 1.a977f6a7 e556) v4 -- ?+0 0x2c926c0 con 0x2ca9840
-96> 2013-03-13 10:05:10.357944 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-95> 2013-03-13 10:05:10.357959 7f9a6c39d700 0 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
-94> 2013-03-13 10:05:10.357972 7f9a6eca3700 2 mds.0.21 boot_start 1: opening snap table
-93> 2013-03-13 10:05:10.357983 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/64223, 111: Connection refused
-92> 2013-03-13 10:05:10.357996 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-91> 2013-03-13 10:05:10.358002 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6806/64507 -- osd_op(mds.0.21:4 mds_snaptable [read 0~0] 1.d90270ad e556) v4 -- ?+0 0x2c92d80 con 0x2ca9b00
-90> 2013-03-13 10:05:10.358012 7f9a6eca3700 2 mds.0.21 boot_start 1: opening mds log
-89> 2013-03-13 10:05:10.358015 7f9a6eca3700 5 mds.0.log open discovering log bounds
-88> 2013-03-13 10:05:10.358020 7f9a6eca3700 1 mds.0.journaler(ro) recover start
-87> 2013-03-13 10:05:10.358023 7f9a6eca3700 1 mds.0.journaler(ro) read_head
-86> 2013-03-13 10:05:10.358046 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6801/64223 -- osd_op(mds.0.21:5 200.00000000 [read 0~0] 1.844f3494 e556) v4 -- ?+0 0x2c92b40 con 0x2ca9840
-85> 2013-03-13 10:05:10.358064 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 11 ==== osd_map(556..556 src has 55..556) v3 ==== 3198+0+0 (1670496509 0 0) 0x2cbe200 con 0x2ca9580
-84> 2013-03-13 10:05:10.358082 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2cbf000 con 0x2ca9580
-83> 2013-03-13 10:05:10.358089 7f9a6eca3700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
-82> 2013-03-13 10:05:10.358118 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/64223, 111: Connection refused
-81> 2013-03-13 10:05:10.358145 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-80> 2013-03-13 10:05:10.358356 7f9a6eca3700 0 mds.0.21 ms_handle_connect on 10.214.136.10:6806/64507
-79> 2013-03-13 10:05:10.403453 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== osd.1 10.214.136.10:6806/64507 1 ==== osd_op_reply(4 mds_snaptable [read 0~41] ondisk = 0) v4 ==== 112+0+41 (654151839 0 2312406677) 0x2c99400 con 0x2ca9b00
-78> 2013-03-13 10:05:10.758292 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/64223, 111: Connection refused
-77> 2013-03-13 10:05:10.758319 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-76> 2013-03-13 10:05:11.384232 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cc2280 sd=18 :36413 s=2 pgs=10 cs=1 l=1).reader couldn't read tag, Success
-75> 2013-03-13 10:05:11.384294 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cc2280 sd=18 :36413 s=2 pgs=10 cs=1 l=1).fault 0: Success
-74> 2013-03-13 10:05:11.384396 7f9a6eca3700 0 mds.0.21 ms_handle_reset on 10.214.136.10:6806/64507
-73> 2013-03-13 10:05:11.384431 7f9a6eca3700 1 mds.0.objecter ms_handle_reset on osd.1
-72> 2013-03-13 10:05:11.384462 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 mark_down 0x2ca9b00 -- pipe dne
-71> 2013-03-13 10:05:11.384512 7f9a6eca3700 10 monclient: renew_subs
-70> 2013-03-13 10:05:11.384517 7f9a6eca3700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-69> 2013-03-13 10:05:11.384521 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=990+,monmap=2+,osdmap=557}) v2 -- ?+0 0x2cbf000 con 0x2ca9580
-68> 2013-03-13 10:05:11.384560 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-67> 2013-03-13 10:05:11.384614 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-66> 2013-03-13 10:05:11.384626 7f9a6c19b700 0 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
-65> 2013-03-13 10:05:11.384645 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-64> 2013-03-13 10:05:11.384652 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-63> 2013-03-13 10:05:11.584803 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-62> 2013-03-13 10:05:11.584827 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-61> 2013-03-13 10:05:11.684911 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 13 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2cbf380 con 0x2ca9580
-60> 2013-03-13 10:05:11.684925 7f9a6eca3700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:05:11.384514 renew after 2013-03-13 10:07:41.384514
-59> 2013-03-13 10:05:11.985004 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-58> 2013-03-13 10:05:11.985047 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-57> 2013-03-13 10:05:12.675927 7f9a6dca1700 10 monclient: tick
-56> 2013-03-13 10:05:12.675955 7f9a6dca1700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 10:04:42.675953)
-55> 2013-03-13 10:05:12.675971 7f9a6dca1700 10 monclient: renew subs? (now: 2013-03-13 10:05:12.675971; renew after: 2013-03-13 10:07:41.384514) -- no
-54> 2013-03-13 10:05:12.785189 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-53> 2013-03-13 10:05:12.785230 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-52> 2013-03-13 10:05:13.857817 7f9a6cb9e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-51> 2013-03-13 10:05:13.857849 7f9a6cb9e700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mdsbeacon(8999/a up:replay seq 2 v989) v2 -- ?+0 0x2cb62c0 con 0x2ca9580
-50> 2013-03-13 10:05:13.858237 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 14 ==== mdsbeacon(8999/a up:replay seq 2 v989) v2 ==== 103+0+0 (1134139146 0 0) 0x2cb6000 con 0x2ca9580
-49> 2013-03-13 10:05:14.385366 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-48> 2013-03-13 10:05:14.385405 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-47> 2013-03-13 10:05:14.859466 7f9a6c39d700 0 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :37065 s=1 pgs=0 cs=0 l=1).connect claims to be 0.0.0.0:6801/6015 not 10.214.136.10:6801/64223 - wrong node!
-46> 2013-03-13 10:05:14.859493 7f9a6c39d700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6801/64223 pipe(0x2cc2500 sd=17 :37065 s=1 pgs=0 cs=0 l=1).fault 107: Transport endpoint is not connected
-45> 2013-03-13 10:05:15.686954 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 15 ==== osd_map(557..557 src has 56..557) v3 ==== 169+0+0 (4265868525 0 0) 0x2cbe000 con 0x2ca9580
-44> 2013-03-13 10:05:15.687018 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 mark_down 0x2ca9840 -- 0x2cc2500
-43> 2013-03-13 10:05:15.687043 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6806/64507 -- osd_op(mds.0.21:1 mds0_inotable [read 0~0] 1.b852b893 RETRY=1 e557) v4 -- ?+0 0x2ccb6c0 con 0x2ca99a0
-42> 2013-03-13 10:05:15.687106 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6806/64507 -- osd_op(mds.0.21:2 mds0_sessionmap [read 0~0] 1.3270c60b RETRY=1 e557) v4 -- ?+0 0x2ccb480 con 0x2ca99a0
-41> 2013-03-13 10:05:15.687118 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6806/64507 -- osd_op(mds.0.21:3 mds_anchortable [read 0~0] 1.a977f6a7 RETRY=1 e557) v4 -- ?+0 0x2ccb240 con 0x2ca99a0
-40> 2013-03-13 10:05:15.687128 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6806/64507 -- osd_op(mds.0.21:5 200.00000000 [read 0~0] 1.844f3494 RETRY=1 e557) v4 -- ?+0 0x2ccb000 con 0x2ca99a0
-39> 2013-03-13 10:05:15.687171 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-38> 2013-03-13 10:05:15.687209 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-37> 2013-03-13 10:05:17.857998 7f9a6cb9e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-36> 2013-03-13 10:05:17.858020 7f9a6cb9e700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mdsbeacon(8999/a up:replay seq 3 v989) v2 -- ?+0 0x2cb6580 con 0x2ca9580
-35> 2013-03-13 10:05:17.858371 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 16 ==== mdsbeacon(8999/a up:replay seq 3 v989) v2 ==== 103+0+0 (984672877 0 0) 0x2cb6b00 con 0x2ca9580
-34> 2013-03-13 10:05:21.858187 7f9a6cb9e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-33> 2013-03-13 10:05:21.858214 7f9a6cb9e700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mdsbeacon(8999/a up:replay seq 4 v989) v2 -- ?+0 0x2cb62c0 con 0x2ca9580
-32> 2013-03-13 10:05:21.858604 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 17 ==== mdsbeacon(8999/a up:replay seq 4 v989) v2 ==== 103+0+0 (1350758057 0 0) 0x2cb6840 con 0x2ca9580
-31> 2013-03-13 10:05:22.087394 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-30> 2013-03-13 10:05:22.087433 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-29> 2013-03-13 10:05:22.676061 7f9a6dca1700 10 monclient: tick
-28> 2013-03-13 10:05:22.676092 7f9a6dca1700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 10:04:52.676090)
-27> 2013-03-13 10:05:22.676104 7f9a6dca1700 10 monclient: renew subs? (now: 2013-03-13 10:05:22.676104; renew after: 2013-03-13 10:07:41.384514) -- no
-26> 2013-03-13 10:05:25.858321 7f9a6cb9e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-25> 2013-03-13 10:05:25.858345 7f9a6cb9e700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mdsbeacon(8999/a up:replay seq 5 v989) v2 -- ?+0 0x2cb6dc0 con 0x2ca9580
-24> 2013-03-13 10:05:25.858739 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 18 ==== mdsbeacon(8999/a up:replay seq 5 v989) v2 ==== 103+0+0 (699110350 0 0) 0x2cb6dc0 con 0x2ca9580
-23> 2013-03-13 10:05:29.858127 7f9a6cb9e700 10 monclient: renew_subs
-22> 2013-03-13 10:05:29.858142 7f9a6cb9e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-21> 2013-03-13 10:05:29.858151 7f9a6cb9e700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=990+,monmap=2+,osdmap=558}) v2 -- ?+0 0x2cbf540 con 0x2ca9580
-20> 2013-03-13 10:05:29.858171 7f9a6cb9e700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6806/64507 -- ping v1 -- ?+0 0x2cbf8c0 con 0x2ca99a0
-19> 2013-03-13 10:05:29.858258 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6806/64507, 111: Connection refused
-18> 2013-03-13 10:05:29.858293 7f9a6c19b700 2 -- 10.214.136.10:6800/5730 >> 10.214.136.10:6806/64507 pipe(0x2cacc80 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-17> 2013-03-13 10:05:29.858444 7f9a6cb9e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-16> 2013-03-13 10:05:29.858451 7f9a6cb9e700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6789/0 -- mdsbeacon(8999/a up:replay seq 6 v989) v2 -- ?+0 0x2cb6b00 con 0x2ca9580
-15> 2013-03-13 10:05:29.940931 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 19 ==== osd_map(558..561 src has 60..561) v3 ==== 1513+0+0 (1472338668 0 0) 0x2cbec00 con 0x2ca9580
-14> 2013-03-13 10:05:29.941052 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 mark_down 0x2ca99a0 -- 0x2cacc80
-13> 2013-03-13 10:05:29.941086 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6801/6015 -- osd_op(mds.0.21:1 mds0_inotable [read 0~0] 1.b852b893 RETRY=2 e561) v4 -- ?+0 0x2ccb900 con 0x2ca9b00
-12> 2013-03-13 10:05:29.941148 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6801/6015 -- osd_op(mds.0.21:2 mds0_sessionmap [read 0~0] 1.3270c60b RETRY=2 e561) v4 -- ?+0 0x2c92480 con 0x2ca9b00
-11> 2013-03-13 10:05:29.941176 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6801/6015 -- osd_op(mds.0.21:3 mds_anchortable [read 0~0] 1.a977f6a7 RETRY=2 e561) v4 -- ?+0 0x2c92900 con 0x2ca9b00
-10> 2013-03-13 10:05:29.941186 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 --> 10.214.136.10:6801/6015 -- osd_op(mds.0.21:5 200.00000000 [read 0~0] 1.844f3494 RETRY=2 e561) v4 -- ?+0 0x2c926c0 con 0x2ca9b00
-9> 2013-03-13 10:05:29.941215 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 20 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x2cbf1c0 con 0x2ca9580
-8> 2013-03-13 10:05:29.941227 7f9a6eca3700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:05:29.858138 renew after 2013-03-13 10:07:59.858138
-7> 2013-03-13 10:05:29.941237 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== mon.0 10.214.136.10:6789/0 21 ==== mdsbeacon(8999/a up:replay seq 6 v989) v2 ==== 103+0+0 (2731545703 0 0) 0x2cb62c0 con 0x2ca9580
-6> 2013-03-13 10:05:29.941503 7f9a6eca3700 0 mds.0.21 ms_handle_connect on 10.214.136.10:6801/6015
-5> 2013-03-13 10:05:32.676237 7f9a6dca1700 10 monclient: tick
-4> 2013-03-13 10:05:32.676262 7f9a6dca1700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 10:05:02.676260)
-3> 2013-03-13 10:05:32.676279 7f9a6dca1700 10 monclient: renew subs? (now: 2013-03-13 10:05:32.676279; renew after: 2013-03-13 10:07:59.858138) -- no
-2> 2013-03-13 10:05:32.835263 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== osd.0 10.214.136.10:6801/6015 1 ==== osd_op_reply(1 mds0_inotable [read 0~29] ondisk = 0) v4 ==== 112+0+29 (646261609 0 2795693049) 0x2c99400 con 0x2ca9b00
-1> 2013-03-13 10:05:32.848134 7f9a6eca3700 1 -- 10.214.136.10:6800/5730 <== osd.0 10.214.136.10:6801/6015 2 ==== osd_op_reply(2 mds0_sessionmap [read 0~1848] ondisk = 0) v4 ==== 114+0+1848 (1846661310 0 1211354084) 0x2cbe000 con 0x2ca9b00
0> 2013-03-13 10:05:32.849786 7f9a6eca3700 -1 *** Caught signal (Aborted) **
in thread 7f9a6eca3700

ceph version 0.58-500-gaf3b163 (af3b16349a49a8aee401e27c1b71fd704b31297c)
1: /usr/bin/ceph-mds() [0x84fa7a]
2: (()+0xfcb0) [0x7f9a73007cb0]
3: (gsignal()+0x35) [0x7f9a716f8425]
4: (abort()+0x17b) [0x7f9a716fbb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9a7204a69d]
6: (()+0xb5846) [0x7f9a72048846]
7: (()+0xb5873) [0x7f9a72048873]
8: (()+0xb596e) [0x7f9a7204896e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7adcd7]
10: (void decode<unsigned long, inodeno_t>(std::map<unsigned long, inodeno_t, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xde) [0x6b2cae]
11: (session_info_t::decode(ceph::buffer::list::iterator&)+0x139) [0x7d7ee9]
12: (SessionMap::decode(ceph::buffer::list::iterator&)+0xb98) [0x6c87e8]
13: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x83) [0x6c8cc3]
14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe38) [0x6f17b8]
15: (MDS::handle_core_message(Message*)+0xae8) [0x4db088]
16: (MDS::_dispatch(Message*)+0x2f) [0x4db24f]
17: (MDS::ms_dispatch(Message*)+0x1db) [0x4dcceb]
18: (DispatchQueue::entry()+0x341) [0x81e1a1]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x79c45d]
20: (()+0x7e9a) [0x7f9a72fffe9a]
21: (clone()+0x6d) [0x7f9a717b5cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.a.log
--- end dump of recent events ---
2013-03-13 10:59:27.367708 7f100058a700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
2013-03-13 10:59:27.758717 7f100058a700 1 mds.0.22 handle_mds_map i am now mds.0.22
2013-03-13 10:59:27.758721 7f100058a700 1 mds.0.22 handle_mds_map state change up:boot --> up:replay
2013-03-13 10:59:27.758724 7f100058a700 1 mds.0.22 replay_start
2013-03-13 10:59:27.758729 7f100058a700 1 mds.0.22 recovery set is
2013-03-13 10:59:27.758730 7f100058a700 1 mds.0.22 need osdmap epoch 580, have 580
2013-03-13 10:59:27.758902 7f0ffdc84700 0 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 10:59:27.759376 7f100058a700 0 mds.0.22 ms_handle_connect on 10.214.136.10:6805/6301
2013-03-13 10:59:29.011489 7f100058a700 0 mds.0.22 ms_handle_reset on 10.214.136.10:6805/6301
2013-03-13 10:59:29.011745 7f0ffd981700 0 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 10:59:47.371040 7f100058a700 0 mds.0.22 ms_handle_connect on 10.214.136.10:6802/26568
2013-03-13 10:59:47.373203 7f100058a700 -1 *** Caught signal (Aborted) **
in thread 7f100058a700

ceph version 0.58-501-g66be33a (66be33a3e783b605e46d735a008d6ba79f58a63d)
1: /usr/bin/ceph-mds() [0x84fa7a]
2: (()+0xfcb0) [0x7f10048eecb0]
3: (gsignal()+0x35) [0x7f1002fdf425]
4: (abort()+0x17b) [0x7f1002fe2b8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f100393169d]
6: (()+0xb5846) [0x7f100392f846]
7: (()+0xb5873) [0x7f100392f873]
8: (()+0xb596e) [0x7f100392f96e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7adcd7]
10: (void decode<unsigned long, inodeno_t>(std::map<unsigned long, inodeno_t, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xde) [0x6b2cae]
11: (session_info_t::decode(ceph::buffer::list::iterator&)+0x139) [0x7d7ee9]
12: (SessionMap::decode(ceph::buffer::list::iterator&)+0xb98) [0x6c87e8]
13: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x83) [0x6c8cc3]
14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe38) [0x6f17b8]
15: (MDS::handle_core_message(Message*)+0xae8) [0x4db088]
16: (MDS::_dispatch(Message*)+0x2f) [0x4db24f]
17: (MDS::ms_dispatch(Message*)+0x1db) [0x4dcceb]
18: (DispatchQueue::entry()+0x341) [0x81e1a1]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x79c45d]
20: (()+0x7e9a) [0x7f10048e6e9a]
21: (clone()+0x6d) [0x7f100309ccbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-189> 2013-03-13 10:59:27.364455 7f1004d0b780 5 asok(0x2433000) register_command perfcounters_dump hook 0x2428010
-188> 2013-03-13 10:59:27.364481 7f1004d0b780 5 asok(0x2433000) register_command 1 hook 0x2428010
-187> 2013-03-13 10:59:27.364486 7f1004d0b780 5 asok(0x2433000) register_command perf dump hook 0x2428010
-186> 2013-03-13 10:59:27.364492 7f1004d0b780 5 asok(0x2433000) register_command perfcounters_schema hook 0x2428010
-185> 2013-03-13 10:59:27.364496 7f1004d0b780 5 asok(0x2433000) register_command 2 hook 0x2428010
-184> 2013-03-13 10:59:27.364500 7f1004d0b780 5 asok(0x2433000) register_command perf schema hook 0x2428010
-183> 2013-03-13 10:59:27.364502 7f1004d0b780 5 asok(0x2433000) register_command config show hook 0x2428010
-182> 2013-03-13 10:59:27.364506 7f1004d0b780 5 asok(0x2433000) register_command config set hook 0x2428010
-181> 2013-03-13 10:59:27.364510 7f1004d0b780 5 asok(0x2433000) register_command log flush hook 0x2428010
-180> 2013-03-13 10:59:27.364513 7f1004d0b780 5 asok(0x2433000) register_command log dump hook 0x2428010
-179> 2013-03-13 10:59:27.364516 7f1004d0b780 5 asok(0x2433000) register_command log reopen hook 0x2428010
-178> 2013-03-13 10:59:27.365760 7f1004d0b780 0 ceph version 0.58-501-g66be33a (66be33a3e783b605e46d735a008d6ba79f58a63d), process ceph-mds, pid 26285
-177> 2013-03-13 10:59:27.366155 7f1004d0b780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/26285 need_addr=1
-176> 2013-03-13 10:59:27.366593 7f1004d0b780 1 finished global_init_daemonize
-175> 2013-03-13 10:59:27.366748 7f1004d0b780 5 asok(0x2433000) init /var/run/ceph/ceph-mds.a.asok
-174> 2013-03-13 10:59:27.366764 7f1004d0b780 5 asok(0x2433000) bind_and_listen /var/run/ceph/ceph-mds.a.asok
-173> 2013-03-13 10:59:27.366800 7f1004d0b780 5 asok(0x2433000) register_command 0 hook 0x2427038
-172> 2013-03-13 10:59:27.366807 7f1004d0b780 5 asok(0x2433000) register_command version hook 0x2427038
-171> 2013-03-13 10:59:27.366812 7f1004d0b780 5 asok(0x2433000) register_command git_version hook 0x2427038
-170> 2013-03-13 10:59:27.366818 7f1004d0b780 5 asok(0x2433000) register_command help hook 0x2428050
-169> 2013-03-13 10:59:27.366863 7f1004d0b780 10 monclient(hunting): build_initial_monmap
-168> 2013-03-13 10:59:27.366856 7f1001d8d700 5 asok(0x2433000) entry start
-167> 2013-03-13 10:59:27.366911 7f1004d0b780 1 -- 0.0.0.0:6800/26285 messenger.start
-166> 2013-03-13 10:59:27.367001 7f1004d0b780 5 adding auth protocol: cephx
-165> 2013-03-13 10:59:27.367009 7f1004d0b780 5 adding auth protocol: cephx
-164> 2013-03-13 10:59:27.367091 7f1004d0b780 1 accepter.accepter.start
-163> 2013-03-13 10:59:27.367118 7f1004d0b780 10 monclient(hunting): init
-162> 2013-03-13 10:59:27.367124 7f1004d0b780 5 adding auth protocol: cephx
-161> 2013-03-13 10:59:27.367125 7f1004d0b780 10 monclient(hunting): auth_supported 2 method cephx
-160> 2013-03-13 10:59:27.367321 7f1004d0b780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-a/keyring
-159> 2013-03-13 10:59:27.367386 7f1004d0b780 10 monclient(hunting): _reopen_session
-158> 2013-03-13 10:59:27.367424 7f1004d0b780 10 monclient(hunting): _pick_new_mon picked mon.a con 0x2449580 addr 10.214.136.10:6789/0
-157> 2013-03-13 10:59:27.367440 7f1004d0b780 10 monclient(hunting): _send_mon_message to mon.a at 10.214.136.10:6789/0
-156> 2013-03-13 10:59:27.367446 7f1004d0b780 1 -- 0.0.0.0:6800/26285 --> 10.214.136.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x2439400 con 0x2449580
-155> 2013-03-13 10:59:27.367459 7f1004d0b780 10 monclient(hunting): renew_subs
-154> 2013-03-13 10:59:27.367594 7f1004d07700 1 -- 10.214.136.10:6800/26285 learned my addr 10.214.136.10:6800/26285
-153> 2013-03-13 10:59:27.367708 7f100058a700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
-152> 2013-03-13 10:59:27.368052 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 1 ==== mon_map v1 ==== 191+0+0 (4269767265 0 0) 0x2439600 con 0x2449580
-151> 2013-03-13 10:59:27.368069 7f100058a700 10 monclient(hunting): handle_monmap mon_map v1
-150> 2013-03-13 10:59:27.368084 7f100058a700 10 monclient(hunting): got monmap 1, mon.a is now rank 0
-149> 2013-03-13 10:59:27.368087 7f100058a700 10 monclient(hunting): dump:
epoch 1
fsid 3b3c3298-665f-4146-bdbf-376298b14e91
last_changed 2013-02-15 08:55:38.400848
created 2013-02-15 08:55:38.400848
0: 10.214.136.10:6789/0 mon.a

-148> 2013-03-13 10:59:27.368119 7f100058a700 1 monclient(hunting): found mon.a
-147> 2013-03-13 10:59:27.368126 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3061460567 0 0) 0x2439a00 con 0x2449580
-146> 2013-03-13 10:59:27.368148 7f100058a700 10 monclient: my global_id is 15098
-145> 2013-03-13 10:59:27.368352 7f100058a700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-144> 2013-03-13 10:59:27.368358 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x2439600 con 0x2449580
-143> 2013-03-13 10:59:27.368671 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (2754354384 0 0) 0x2439800 con 0x2449580
-142> 2013-03-13 10:59:27.368746 7f100058a700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-141> 2013-03-13 10:59:27.368751 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2439a00 con 0x2449580
-140> 2013-03-13 10:59:27.369038 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (1336156894 0 0) 0x2439c00 con 0x2449580
-139> 2013-03-13 10:59:27.369074 7f100058a700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-138> 2013-03-13 10:59:27.369078 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x244d700 con 0x2449580
-137> 2013-03-13 10:59:27.369098 7f100058a700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-03-13 10:58:57.369097)
-136> 2013-03-13 10:59:27.369106 7f100058a700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-135> 2013-03-13 10:59:27.369110 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2439800 con 0x2449580
-134> 2013-03-13 10:59:27.369123 7f1004d0b780 5 monclient: authenticate success, global_id 15098
-133> 2013-03-13 10:59:27.369134 7f1004d0b780 10 monclient: wait_auth_rotating waiting (until 2013-03-13 10:59:57.369133)
-132> 2013-03-13 10:59:27.369228 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 5 ==== mon_map v1 ==== 191+0+0 (4269767265 0 0) 0x2439600 con 0x2449580
-131> 2013-03-13 10:59:27.369236 7f100058a700 10 monclient: handle_monmap mon_map v1
-130> 2013-03-13 10:59:27.369241 7f100058a700 10 monclient: got monmap 1, mon.a is now rank 0
-129> 2013-03-13 10:59:27.369242 7f100058a700 10 monclient: dump:
epoch 1
fsid 3b3c3298-665f-4146-bdbf-376298b14e91
last_changed 2013-02-15 08:55:38.400848
created 2013-02-15 08:55:38.400848
0: 10.214.136.10:6789/0 mon.a

-128> 2013-03-13 10:59:27.369276 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x244da80 con 0x2449580
-127> 2013-03-13 10:59:27.369298 7f100058a700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:59:27.367462 renew after 2013-03-13 11:01:57.367462
-126> 2013-03-13 10:59:27.369412 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1379138577 0 0) 0x2439400 con 0x2449580
-125> 2013-03-13 10:59:27.369472 7f100058a700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 10:58:57.369471)
-124> 2013-03-13 10:59:27.369485 7f1004d0b780 10 monclient: wait_auth_rotating done
-123> 2013-03-13 10:59:27.369500 7f1004d0b780 5 asok(0x2433000) register_command objecter_requests hook 0x24280d0
-122> 2013-03-13 10:59:27.369546 7f1004d0b780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-121> 2013-03-13 10:59:27.369554 7f1004d0b780 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mdsbeacon(15098/a up:boot seq 1 v0) v2 -- ?+0 0x2456000 con 0x2449580
-120> 2013-03-13 10:59:27.369572 7f1004d0b780 10 monclient: renew_subs
-119> 2013-03-13 10:59:27.369577 7f1004d0b780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-118> 2013-03-13 10:59:27.369581 7f1004d0b780 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x244d540 con 0x2449580
-117> 2013-03-13 10:59:27.369590 7f1004d0b780 10 monclient: renew_subs
-116> 2013-03-13 10:59:27.369593 7f1004d0b780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-115> 2013-03-13 10:59:27.369597 7f1004d0b780 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x244de00 con 0x2449580
-114> 2013-03-13 10:59:27.758292 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 8 ==== osd_map(580..580 src has 79..580) v3 ==== 3086+0+0 (3280488015 0 0) 0x245e600 con 0x2449580
-113> 2013-03-13 10:59:27.758440 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x244d540 con 0x2449580
-112> 2013-03-13 10:59:27.758451 7f100058a700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:59:27.369575 renew after 2013-03-13 11:01:57.369575
-111> 2013-03-13 10:59:27.758641 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 10 ==== mdsmap(e 1198) v1 ==== 583+0+0 (3325242304 0 0) 0x245e400 con 0x2449580
-110> 2013-03-13 10:59:27.758674 7f100058a700 5 mds.-1.0 handle_mds_map epoch 1198 from mon.0
-109> 2013-03-13 10:59:27.758717 7f100058a700 1 mds.0.22 handle_mds_map i am now mds.0.22
-108> 2013-03-13 10:59:27.758721 7f100058a700 1 mds.0.22 handle_mds_map state change up:boot --> up:replay
-107> 2013-03-13 10:59:27.758724 7f100058a700 1 mds.0.22 replay_start
-106> 2013-03-13 10:59:27.758729 7f100058a700 1 mds.0.22 recovery set is
-105> 2013-03-13 10:59:27.758730 7f100058a700 1 mds.0.22 need osdmap epoch 580, have 580
-104> 2013-03-13 10:59:27.758734 7f100058a700 2 mds.0.22 boot_start 1: opening inotable
-103> 2013-03-13 10:59:27.758827 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6801/6015 -- osd_op(mds.0.22:1 mds0_inotable [read 0~0] 1.b852b893 e580) v4 -- ?+0 0x2432480 con 0x2449840
-102> 2013-03-13 10:59:27.758847 7f100058a700 2 mds.0.22 boot_start 1: opening sessionmap
-101> 2013-03-13 10:59:27.758860 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6801/6015 -- osd_op(mds.0.22:2 mds0_sessionmap [read 0~0] 1.3270c60b e580) v4 -- ?+0 0x2432900 con 0x2449840
-100> 2013-03-13 10:59:27.758868 7f100058a700 2 mds.0.22 boot_start 1: opening anchor table
-99> 2013-03-13 10:59:27.758849 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/6015, 111: Connection refused
-98> 2013-03-13 10:59:27.758886 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6801/6015 -- osd_op(mds.0.22:3 mds_anchortable [read 0~0] 1.a977f6a7 e580) v4 -- ?+0 0x24326c0 con 0x2449840
-97> 2013-03-13 10:59:27.758890 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-96> 2013-03-13 10:59:27.758902 7f0ffdc84700 0 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
-95> 2013-03-13 10:59:27.758915 7f100058a700 2 mds.0.22 boot_start 1: opening snap table
-94> 2013-03-13 10:59:27.758927 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/6015, 111: Connection refused
-93> 2013-03-13 10:59:27.758938 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-92> 2013-03-13 10:59:27.758948 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6805/6301 -- osd_op(mds.0.22:4 mds_snaptable [read 0~0] 1.d90270ad e580) v4 -- ?+0 0x2432d80 con 0x2449b00
-91> 2013-03-13 10:59:27.758959 7f100058a700 2 mds.0.22 boot_start 1: opening mds log
-90> 2013-03-13 10:59:27.758965 7f100058a700 5 mds.0.log open discovering log bounds
-89> 2013-03-13 10:59:27.758972 7f100058a700 1 mds.0.journaler(ro) recover start
-88> 2013-03-13 10:59:27.758975 7f100058a700 1 mds.0.journaler(ro) read_head
-87> 2013-03-13 10:59:27.758988 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6801/6015 -- osd_op(mds.0.22:5 200.00000000 [read 0~0] 1.844f3494 e580) v4 -- ?+0 0x2432b40 con 0x2449840
-86> 2013-03-13 10:59:27.759008 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 11 ==== osd_map(580..580 src has 79..580) v3 ==== 3086+0+0 (3280488015 0 0) 0x245e200 con 0x2449580
-85> 2013-03-13 10:59:27.759028 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x244d700 con 0x2449580
-84> 2013-03-13 10:59:27.759035 7f100058a700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
-83> 2013-03-13 10:59:27.759042 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/6015, 111: Connection refused
-82> 2013-03-13 10:59:27.759066 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-81> 2013-03-13 10:59:27.759376 7f100058a700 0 mds.0.22 ms_handle_connect on 10.214.136.10:6805/6301
-80> 2013-03-13 10:59:28.159219 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/6015, 111: Connection refused
-79> 2013-03-13 10:59:28.159240 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-78> 2013-03-13 10:59:28.556768 7f100058a700 1 -- 10.214.136.10:6800/26285 <== osd.1 10.214.136.10:6805/6301 1 ==== osd_op_reply(4 mds_snaptable [read 0~41] ondisk = 0) v4 ==== 112+0+41 (2575286692 0 2312406677) 0x2439400 con 0x2449b00
-77> 2013-03-13 10:59:28.959378 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/6015, 111: Connection refused
-76> 2013-03-13 10:59:28.959420 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-75> 2013-03-13 10:59:29.011320 7f0ffda82700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x2460280 sd=18 :42833 s=2 pgs=68 cs=1 l=1).reader couldn't read tag, Success
-74> 2013-03-13 10:59:29.011394 7f0ffda82700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x2460280 sd=18 :42833 s=2 pgs=68 cs=1 l=1).fault 0: Success
-73> 2013-03-13 10:59:29.011489 7f100058a700 0 mds.0.22 ms_handle_reset on 10.214.136.10:6805/6301
-72> 2013-03-13 10:59:29.011500 7f100058a700 1 mds.0.objecter ms_handle_reset on osd.1
-71> 2013-03-13 10:59:29.011504 7f100058a700 1 -- 10.214.136.10:6800/26285 mark_down 0x2449b00 -- pipe dne
-70> 2013-03-13 10:59:29.011611 7f100058a700 10 monclient: renew_subs
-69> 2013-03-13 10:59:29.011620 7f100058a700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-68> 2013-03-13 10:59:29.011628 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=1199+,monmap=2+,osdmap=581}) v2 -- ?+0 0x244d700 con 0x2449580
-67> 2013-03-13 10:59:29.011668 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=19 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-66> 2013-03-13 10:59:29.011734 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-65> 2013-03-13 10:59:29.011745 7f0ffd981700 0 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault
-64> 2013-03-13 10:59:29.011766 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-63> 2013-03-13 10:59:29.011774 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-62> 2013-03-13 10:59:29.011880 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 13 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x244d540 con 0x2449580
-61> 2013-03-13 10:59:29.011909 7f100058a700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:59:29.011616 renew after 2013-03-13 11:01:59.011616
-60> 2013-03-13 10:59:29.211875 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-59> 2013-03-13 10:59:29.211904 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-58> 2013-03-13 10:59:29.612037 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-57> 2013-03-13 10:59:29.612064 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-56> 2013-03-13 10:59:30.367495 7f0fff588700 10 monclient: tick
-55> 2013-03-13 10:59:30.367521 7f0fff588700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 10:59:00.367519)
-54> 2013-03-13 10:59:30.367533 7f0fff588700 10 monclient: renew subs? (now: 2013-03-13 10:59:30.367533; renew after: 2013-03-13 11:01:59.011616) -- no
-53> 2013-03-13 10:59:30.412216 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-52> 2013-03-13 10:59:30.412247 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-51> 2013-03-13 10:59:30.559576 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/6015, 111: Connection refused
-50> 2013-03-13 10:59:30.559603 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-49> 2013-03-13 10:59:31.369705 7f0ffe485700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-48> 2013-03-13 10:59:31.369739 7f0ffe485700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mdsbeacon(15098/a up:replay seq 2 v1198) v2 -- ?+0 0x24562c0 con 0x2449580
-47> 2013-03-13 10:59:31.370148 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 14 ==== mdsbeacon(15098/a up:replay seq 2 v1198) v2 ==== 103+0+0 (2710238940 0 0) 0x2456000 con 0x2449580
-46> 2013-03-13 10:59:32.012414 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-45> 2013-03-13 10:59:32.012456 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-44> 2013-03-13 10:59:33.759734 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6801/6015, 111: Connection refused
-43> 2013-03-13 10:59:33.759774 7f0ffdc84700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6801/6015 pipe(0x2460500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-42> 2013-03-13 10:59:35.212568 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-41> 2013-03-13 10:59:35.212611 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-40> 2013-03-13 10:59:35.369844 7f0ffe485700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-39> 2013-03-13 10:59:35.369867 7f0ffe485700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mdsbeacon(15098/a up:replay seq 3 v1198) v2 -- ?+0 0x2456580 con 0x2449580
-38> 2013-03-13 10:59:35.494715 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 15 ==== osd_map(581..581 src has 80..581) v3 ==== 169+0+0 (2082667969 0 0) 0x245e000 con 0x2449580
-37> 2013-03-13 10:59:35.494787 7f100058a700 1 -- 10.214.136.10:6800/26285 mark_down 0x2449840 -- 0x2460500
-36> 2013-03-13 10:59:35.494814 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6805/6301 -- osd_op(mds.0.22:1 mds0_inotable [read 0~0] 1.b852b893 RETRY=1 e581) v4 -- ?+0 0x246d6c0 con 0x24499a0
-35> 2013-03-13 10:59:35.494830 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6805/6301 -- osd_op(mds.0.22:2 mds0_sessionmap [read 0~0] 1.3270c60b RETRY=1 e581) v4 -- ?+0 0x246d480 con 0x24499a0
-34> 2013-03-13 10:59:35.494838 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6805/6301 -- osd_op(mds.0.22:3 mds_anchortable [read 0~0] 1.a977f6a7 RETRY=1 e581) v4 -- ?+0 0x246d240 con 0x24499a0
-33> 2013-03-13 10:59:35.494846 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6805/6301 -- osd_op(mds.0.22:5 200.00000000 [read 0~0] 1.844f3494 RETRY=1 e581) v4 -- ?+0 0x246d000 con 0x24499a0
-32> 2013-03-13 10:59:35.494925 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-31> 2013-03-13 10:59:35.494962 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-30> 2013-03-13 10:59:35.495030 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 16 ==== mdsbeacon(15098/a up:replay seq 3 v1198) v2 ==== 103+0+0 (3634596795 0 0) 0x2456b00 con 0x2449580
-29> 2013-03-13 10:59:39.370033 7f0ffe485700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-28> 2013-03-13 10:59:39.370055 7f0ffe485700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mdsbeacon(15098/a up:replay seq 4 v1198) v2 -- ?+0 0x24562c0 con 0x2449580
-27> 2013-03-13 10:59:39.370420 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 17 ==== mdsbeacon(15098/a up:replay seq 4 v1198) v2 ==== 103+0+0 (2995883903 0 0) 0x2456840 con 0x2449580
-26> 2013-03-13 10:59:40.367674 7f0fff588700 10 monclient: tick
-25> 2013-03-13 10:59:40.367697 7f0fff588700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 10:59:10.367695)
-24> 2013-03-13 10:59:40.367712 7f0fff588700 10 monclient: renew subs? (now: 2013-03-13 10:59:40.367712; renew after: 2013-03-13 11:01:59.011616) -- no
-23> 2013-03-13 10:59:43.370207 7f0ffe485700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-22> 2013-03-13 10:59:43.370230 7f0ffe485700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mdsbeacon(15098/a up:replay seq 5 v1198) v2 -- ?+0 0x2456dc0 con 0x2449580
-21> 2013-03-13 10:59:43.600369 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 18 ==== mdsbeacon(15098/a up:replay seq 5 v1198) v2 ==== 103+0+0 (3417895448 0 0) 0x2456dc0 con 0x2449580
-20> 2013-03-13 10:59:47.370013 7f0ffe485700 10 monclient: renew_subs
-19> 2013-03-13 10:59:47.370036 7f0ffe485700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-18> 2013-03-13 10:59:47.370047 7f0ffe485700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=1199+,monmap=2+,osdmap=582}) v2 -- ?+0 0x246e000 con 0x2449580
-17> 2013-03-13 10:59:47.370068 7f0ffe485700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6805/6301 -- ping v1 -- ?+0 0x246e380 con 0x24499a0
-16> 2013-03-13 10:59:47.370166 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6805/6301, 111: Connection refused
-15> 2013-03-13 10:59:47.370200 7f0ffd981700 2 -- 10.214.136.10:6800/26285 >> 10.214.136.10:6805/6301 pipe(0x244cc80 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-14> 2013-03-13 10:59:47.370310 7f0ffe485700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-13> 2013-03-13 10:59:47.370315 7f0ffe485700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6789/0 -- mdsbeacon(15098/a up:replay seq 6 v1198) v2 -- ?+0 0x2456b00 con 0x2449580
-12> 2013-03-13 10:59:47.370576 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 19 ==== osd_map(582..585 src has 84..585) v3 ==== 1505+0+0 (252999459 0 0) 0x245ec00 con 0x2449580
-11> 2013-03-13 10:59:47.370667 7f100058a700 1 -- 10.214.136.10:6800/26285 mark_down 0x24499a0 -- 0x244cc80
-10> 2013-03-13 10:59:47.370695 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6802/26568 -- osd_op(mds.0.22:1 mds0_inotable [read 0~0] 1.b852b893 RETRY=2 e585) v4 -- ?+0 0x246d900 con 0x2449c60
-9> 2013-03-13 10:59:47.370732 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6802/26568 -- osd_op(mds.0.22:2 mds0_sessionmap [read 0~0] 1.3270c60b RETRY=2 e585) v4 -- ?+0 0x2432480 con 0x2449c60
-8> 2013-03-13 10:59:47.370742 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6802/26568 -- osd_op(mds.0.22:3 mds_anchortable [read 0~0] 1.a977f6a7 RETRY=2 e585) v4 -- ?+0 0x2432900 con 0x2449c60
-7> 2013-03-13 10:59:47.370751 7f100058a700 1 -- 10.214.136.10:6800/26285 --> 10.214.136.10:6802/26568 -- osd_op(mds.0.22:5 200.00000000 [read 0~0] 1.844f3494 RETRY=2 e585) v4 -- ?+0 0x24326c0 con 0x2449c60
-6> 2013-03-13 10:59:47.370767 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 20 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x244da80 con 0x2449580
-5> 2013-03-13 10:59:47.370776 7f100058a700 10 monclient: handle_subscribe_ack sent 2013-03-13 10:59:47.370028 renew after 2013-03-13 11:02:17.370028
-4> 2013-03-13 10:59:47.370785 7f100058a700 1 -- 10.214.136.10:6800/26285 <== mon.0 10.214.136.10:6789/0 21 ==== mdsbeacon(15098/a up:replay seq 6 v1198) v2 ==== 103+0+0 (1086551473 0 0) 0x24562c0 con 0x2449580
-3> 2013-03-13 10:59:47.371040 7f100058a700 0 mds.0.22 ms_handle_connect on 10.214.136.10:6802/26568
-2> 2013-03-13 10:59:47.371596 7f100058a700 1 -- 10.214.136.10:6800/26285 <== osd.0 10.214.136.10:6802/26568 1 ==== osd_op_reply(1 mds0_inotable [read 0~29] ondisk = 0) v4 ==== 112+0+29 (937169090 0 2795693049) 0x2439400 con 0x2449c60
-1> 2013-03-13 10:59:47.371639 7f100058a700 1 -- 10.214.136.10:6800/26285 <== osd.0 10.214.136.10:6802/26568 2 ==== osd_op_reply(2 mds0_sessionmap [read 0~1848] ondisk = 0) v4 ==== 114+0+1848 (2135485205 0 1211354084) 0x245e000 con 0x2449c60
0> 2013-03-13 10:59:47.373203 7f100058a700 -1 *** Caught signal (Aborted) **
in thread 7f100058a700

ceph version 0.58-501-g66be33a (66be33a3e783b605e46d735a008d6ba79f58a63d)
1: /usr/bin/ceph-mds() [0x84fa7a]
2: (()+0xfcb0) [0x7f10048eecb0]
3: (gsignal()+0x35) [0x7f1002fdf425]
4: (abort()+0x17b) [0x7f1002fe2b8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f100393169d]
6: (()+0xb5846) [0x7f100392f846]
7: (()+0xb5873) [0x7f100392f873]
8: (()+0xb596e) [0x7f100392f96e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7adcd7]
10: (void decode<unsigned long, inodeno_t>(std::map<unsigned long, inodeno_t, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xde) [0x6b2cae]
11: (session_info_t::decode(ceph::buffer::list::iterator&)+0x139) [0x7d7ee9]
12: (SessionMap::decode(ceph::buffer::list::iterator&)+0xb98) [0x6c87e8]
13: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x83) [0x6c8cc3]
14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe38) [0x6f17b8]
15: (MDS::handle_core_message(Message*)+0xae8) [0x4db088]
16: (MDS::_dispatch(Message*)+0x2f) [0x4db24f]
17: (MDS::ms_dispatch(Message*)+0x1db) [0x4dcceb]
18: (DispatchQueue::entry()+0x341) [0x81e1a1]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x79c45d]
20: (()+0x7e9a) [0x7f10048e6e9a]
21: (clone()+0x6d) [0x7f100309ccbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.a.log
--- end dump of recent events ---
2013-03-13 11:07:32.482543 7fe08b673700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
2013-03-13 11:07:32.845161 7fe08b673700 1 mds.0.23 handle_mds_map i am now mds.0.23
2013-03-13 11:07:32.845165 7fe08b673700 1 mds.0.23 handle_mds_map state change up:boot --> up:replay
2013-03-13 11:07:32.845168 7fe08b673700 1 mds.0.23 replay_start
2013-03-13 11:07:32.845172 7fe08b673700 1 mds.0.23 recovery set is
2013-03-13 11:07:32.845173 7fe08b673700 1 mds.0.23 need osdmap epoch 586, have 586
2013-03-13 11:07:32.845385 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 11:07:32.845450 7fe088c6c700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
2013-03-13 11:07:37.331995 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56092 s=1 pgs=0 cs=0 l=1).connect claims to be 0.0.0.0:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
2013-03-13 11:07:37.732394 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56106 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
2013-03-13 11:07:38.532740 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56109 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
2013-03-13 11:07:40.133049 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56116 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
2013-03-13 11:07:43.369741 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56122 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
2013-03-13 11:07:47.484833 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56129 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
2013-03-13 11:07:47.485704 7fe08b673700 0 mds.0.23 ms_handle_connect on 10.214.136.10:6801/40956
2013-03-13 11:07:47.486085 7fe08b673700 0 mds.0.23 ms_handle_connect on 10.214.136.10:6804/41191
2013-03-13 11:07:53.921086 7fe08b673700 -1 *** Caught signal (Aborted) **
in thread 7fe08b673700

ceph version 0.58-501-g66be33a (66be33a3e783b605e46d735a008d6ba79f58a63d)
1: /usr/bin/ceph-mds() [0x84fa7a]
2: (()+0xfcb0) [0x7fe08f9d7cb0]
3: (gsignal()+0x35) [0x7fe08e0c8425]
4: (abort()+0x17b) [0x7fe08e0cbb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fe08ea1a69d]
6: (()+0xb5846) [0x7fe08ea18846]
7: (()+0xb5873) [0x7fe08ea18873]
8: (()+0xb596e) [0x7fe08ea1896e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7adcd7]
10: (void decode<unsigned long, inodeno_t>(std::map<unsigned long, inodeno_t, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xde) [0x6b2cae]
11: (session_info_t::decode(ceph::buffer::list::iterator&)+0x139) [0x7d7ee9]
12: (SessionMap::decode(ceph::buffer::list::iterator&)+0xb98) [0x6c87e8]
13: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x83) [0x6c8cc3]
14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe38) [0x6f17b8]
15: (MDS::handle_core_message(Message*)+0xae8) [0x4db088]
16: (MDS::_dispatch(Message*)+0x2f) [0x4db24f]
17: (MDS::ms_dispatch(Message*)+0x1db) [0x4dcceb]
18: (DispatchQueue::entry()+0x341) [0x81e1a1]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x79c45d]
20: (()+0x7e9a) [0x7fe08f9cfe9a]
21: (clone()+0x6d) [0x7fe08e185cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-178> 2013-03-13 11:07:32.479191 7fe08fdf4780 5 asok(0x1cd4000) register_command perfcounters_dump hook 0x1cc9010
-177> 2013-03-13 11:07:32.479214 7fe08fdf4780 5 asok(0x1cd4000) register_command 1 hook 0x1cc9010
-176> 2013-03-13 11:07:32.479217 7fe08fdf4780 5 asok(0x1cd4000) register_command perf dump hook 0x1cc9010
-175> 2013-03-13 11:07:32.479224 7fe08fdf4780 5 asok(0x1cd4000) register_command perfcounters_schema hook 0x1cc9010
-174> 2013-03-13 11:07:32.479228 7fe08fdf4780 5 asok(0x1cd4000) register_command 2 hook 0x1cc9010
-173> 2013-03-13 11:07:32.479232 7fe08fdf4780 5 asok(0x1cd4000) register_command perf schema hook 0x1cc9010
-172> 2013-03-13 11:07:32.479235 7fe08fdf4780 5 asok(0x1cd4000) register_command config show hook 0x1cc9010
-171> 2013-03-13 11:07:32.479238 7fe08fdf4780 5 asok(0x1cd4000) register_command config set hook 0x1cc9010
-170> 2013-03-13 11:07:32.479241 7fe08fdf4780 5 asok(0x1cd4000) register_command log flush hook 0x1cc9010
-169> 2013-03-13 11:07:32.479243 7fe08fdf4780 5 asok(0x1cd4000) register_command log dump hook 0x1cc9010
-168> 2013-03-13 11:07:32.479247 7fe08fdf4780 5 asok(0x1cd4000) register_command log reopen hook 0x1cc9010
-167> 2013-03-13 11:07:32.480551 7fe08fdf4780 0 ceph version 0.58-501-g66be33a (66be33a3e783b605e46d735a008d6ba79f58a63d), process ceph-mds, pid 40709
-166> 2013-03-13 11:07:32.480944 7fe08fdf4780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/40709 need_addr=1
-165> 2013-03-13 11:07:32.481392 7fe08fdf4780 1 finished global_init_daemonize
-164> 2013-03-13 11:07:32.481552 7fe08fdf4780 5 asok(0x1cd4000) init /var/run/ceph/ceph-mds.a.asok
-163> 2013-03-13 11:07:32.481568 7fe08fdf4780 5 asok(0x1cd4000) bind_and_listen /var/run/ceph/ceph-mds.a.asok
-162> 2013-03-13 11:07:32.481603 7fe08fdf4780 5 asok(0x1cd4000) register_command 0 hook 0x1cc8038
-161> 2013-03-13 11:07:32.481610 7fe08fdf4780 5 asok(0x1cd4000) register_command version hook 0x1cc8038
-160> 2013-03-13 11:07:32.481614 7fe08fdf4780 5 asok(0x1cd4000) register_command git_version hook 0x1cc8038
-159> 2013-03-13 11:07:32.481623 7fe08fdf4780 5 asok(0x1cd4000) register_command help hook 0x1cc9050
-158> 2013-03-13 11:07:32.481667 7fe08fdf4780 10 monclient(hunting): build_initial_monmap
-157> 2013-03-13 11:07:32.481661 7fe08ce76700 5 asok(0x1cd4000) entry start
-156> 2013-03-13 11:07:32.481716 7fe08fdf4780 1 -- 0.0.0.0:6800/40709 messenger.start
-155> 2013-03-13 11:07:32.481822 7fe08fdf4780 5 adding auth protocol: cephx
-154> 2013-03-13 11:07:32.481832 7fe08fdf4780 5 adding auth protocol: cephx
-153> 2013-03-13 11:07:32.481908 7fe08fdf4780 1 accepter.accepter.start
-152> 2013-03-13 11:07:32.481937 7fe08fdf4780 10 monclient(hunting): init
-151> 2013-03-13 11:07:32.481943 7fe08fdf4780 5 adding auth protocol: cephx
-150> 2013-03-13 11:07:32.481945 7fe08fdf4780 10 monclient(hunting): auth_supported 2 method cephx
-149> 2013-03-13 11:07:32.482134 7fe08fdf4780 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-a/keyring
-148> 2013-03-13 11:07:32.482196 7fe08fdf4780 10 monclient(hunting): _reopen_session
-147> 2013-03-13 11:07:32.482243 7fe08fdf4780 10 monclient(hunting): _pick_new_mon picked mon.a con 0x1cea580 addr 10.214.136.10:6789/0
-146> 2013-03-13 11:07:32.482259 7fe08fdf4780 10 monclient(hunting): _send_mon_message to mon.a at 10.214.136.10:6789/0
-145> 2013-03-13 11:07:32.482266 7fe08fdf4780 1 -- 0.0.0.0:6800/40709 --> 10.214.136.10:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1cda400 con 0x1cea580
-144> 2013-03-13 11:07:32.482278 7fe08fdf4780 10 monclient(hunting): renew_subs
-143> 2013-03-13 11:07:32.482407 7fe08fdf0700 1 -- 10.214.136.10:6800/40709 learned my addr 10.214.136.10:6800/40709
-142> 2013-03-13 11:07:32.482543 7fe08b673700 0 mds.-1.0 ms_handle_connect on 10.214.136.10:6789/0
-141> 2013-03-13 11:07:32.482880 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 1 ==== mon_map v1 ==== 191+0+0 (4269767265 0 0) 0x1cda600 con 0x1cea580
-140> 2013-03-13 11:07:32.482906 7fe08b673700 10 monclient(hunting): handle_monmap mon_map v1
-139> 2013-03-13 11:07:32.482914 7fe08b673700 10 monclient(hunting): got monmap 1, mon.a is now rank 0
-138> 2013-03-13 11:07:32.482917 7fe08b673700 10 monclient(hunting): dump:
epoch 1
fsid 3b3c3298-665f-4146-bdbf-376298b14e91
last_changed 2013-02-15 08:55:38.400848
created 2013-02-15 08:55:38.400848
0: 10.214.136.10:6789/0 mon.a

-137> 2013-03-13 11:07:32.482947 7fe08b673700 1 monclient(hunting): found mon.a
-136> 2013-03-13 11:07:32.482954 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3501126719 0 0) 0x1cdaa00 con 0x1cea580
-135> 2013-03-13 11:07:32.482977 7fe08b673700 10 monclient: my global_id is 16098
-134> 2013-03-13 11:07:32.483180 7fe08b673700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-133> 2013-03-13 11:07:32.483187 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1cda600 con 0x1cea580
-132> 2013-03-13 11:07:32.483445 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (1355384200 0 0) 0x1cda800 con 0x1cea580
-131> 2013-03-13 11:07:32.483515 7fe08b673700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-130> 2013-03-13 11:07:32.483520 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1cdaa00 con 0x1cea580
-129> 2013-03-13 11:07:32.483801 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 580+0+0 (1382595171 0 0) 0x1cdac00 con 0x1cea580
-128> 2013-03-13 11:07:32.483838 7fe08b673700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-127> 2013-03-13 11:07:32.483843 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1cee700 con 0x1cea580
-126> 2013-03-13 11:07:32.483862 7fe08b673700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-03-13 11:07:02.483861)
-125> 2013-03-13 11:07:32.483871 7fe08b673700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-124> 2013-03-13 11:07:32.483875 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1cda800 con 0x1cea580
-123> 2013-03-13 11:07:32.483891 7fe08fdf4780 5 monclient: authenticate success, global_id 16098
-122> 2013-03-13 11:07:32.483904 7fe08fdf4780 10 monclient: wait_auth_rotating waiting (until 2013-03-13 11:08:02.483903)
-121> 2013-03-13 11:07:32.483991 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 5 ==== mon_map v1 ==== 191+0+0 (4269767265 0 0) 0x1cda600 con 0x1cea580
-120> 2013-03-13 11:07:32.484003 7fe08b673700 10 monclient: handle_monmap mon_map v1
-119> 2013-03-13 11:07:32.484009 7fe08b673700 10 monclient: got monmap 1, mon.a is now rank 0
-118> 2013-03-13 11:07:32.484010 7fe08b673700 10 monclient: dump:
epoch 1
fsid 3b3c3298-665f-4146-bdbf-376298b14e91
last_changed 2013-02-15 08:55:38.400848
created 2013-02-15 08:55:38.400848
0: 10.214.136.10:6789/0 mon.a

-117> 2013-03-13 11:07:32.484025 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x1ceea80 con 0x1cea580
-116> 2013-03-13 11:07:32.484032 7fe08b673700 10 monclient: handle_subscribe_ack sent 2013-03-13 11:07:32.482281 renew after 2013-03-13 11:10:02.482281
-115> 2013-03-13 11:07:32.484128 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 7 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (191859911 0 0) 0x1cda400 con 0x1cea580
-114> 2013-03-13 11:07:32.484166 7fe08b673700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 11:07:02.484165)
-113> 2013-03-13 11:07:32.484179 7fe08fdf4780 10 monclient: wait_auth_rotating done
-112> 2013-03-13 11:07:32.484196 7fe08fdf4780 5 asok(0x1cd4000) register_command objecter_requests hook 0x1cc90d0
-111> 2013-03-13 11:07:32.484248 7fe08fdf4780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-110> 2013-03-13 11:07:32.484257 7fe08fdf4780 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mdsbeacon(16098/a up:boot seq 1 v0) v2 -- ?+0 0x1cf7000 con 0x1cea580
-109> 2013-03-13 11:07:32.484275 7fe08fdf4780 10 monclient: renew_subs
-108> 2013-03-13 11:07:32.484280 7fe08fdf4780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-107> 2013-03-13 11:07:32.484284 7fe08fdf4780 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0x1cee540 con 0x1cea580
-106> 2013-03-13 11:07:32.484292 7fe08fdf4780 10 monclient: renew_subs
-105> 2013-03-13 11:07:32.484295 7fe08fdf4780 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-104> 2013-03-13 11:07:32.484298 7fe08fdf4780 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=0+,monmap=2+,osdmap=0}) v2 -- ?+0 0x1ceee00 con 0x1cea580
-103> 2013-03-13 11:07:32.844959 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 8 ==== osd_map(586..586 src has 85..586) v3 ==== 3230+0+0 (3173237601 0 0) 0x1cff600 con 0x1cea580
-102> 2013-03-13 11:07:32.845100 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x1cee540 con 0x1cea580
-101> 2013-03-13 11:07:32.845110 7fe08b673700 10 monclient: handle_subscribe_ack sent 2013-03-13 11:07:32.484278 renew after 2013-03-13 11:10:02.484278
-100> 2013-03-13 11:07:32.845121 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 10 ==== mdsmap(e 1219) v1 ==== 583+0+0 (2255927552 0 0) 0x1cff400 con 0x1cea580
-99> 2013-03-13 11:07:32.845131 7fe08b673700 5 mds.-1.0 handle_mds_map epoch 1219 from mon.0
-98> 2013-03-13 11:07:32.845161 7fe08b673700 1 mds.0.23 handle_mds_map i am now mds.0.23
-97> 2013-03-13 11:07:32.845165 7fe08b673700 1 mds.0.23 handle_mds_map state change up:boot --> up:replay
-96> 2013-03-13 11:07:32.845168 7fe08b673700 1 mds.0.23 replay_start
-95> 2013-03-13 11:07:32.845172 7fe08b673700 1 mds.0.23 recovery set is
-94> 2013-03-13 11:07:32.845173 7fe08b673700 1 mds.0.23 need osdmap epoch 586, have 586
-93> 2013-03-13 11:07:32.845177 7fe08b673700 2 mds.0.23 boot_start 1: opening inotable
-92> 2013-03-13 11:07:32.845260 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6802/26568 -- osd_op(mds.0.23:1 mds0_inotable [read 0~0] 1.b852b893 e586) v4 -- ?+0 0x1cd3480 con 0x1cea840
-91> 2013-03-13 11:07:32.845280 7fe08b673700 2 mds.0.23 boot_start 1: opening sessionmap
-90> 2013-03-13 11:07:32.845293 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6802/26568 -- osd_op(mds.0.23:2 mds0_sessionmap [read 0~0] 1.3270c60b e586) v4 -- ?+0 0x1cd3900 con 0x1cea840
-89> 2013-03-13 11:07:32.845301 7fe08b673700 2 mds.0.23 boot_start 1: opening anchor table
-88> 2013-03-13 11:07:32.845313 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6802/26568 -- osd_op(mds.0.23:3 mds_anchortable [read 0~0] 1.a977f6a7 e586) v4 -- ?+0 0x1cd36c0 con 0x1cea840
-87> 2013-03-13 11:07:32.845320 7fe08b673700 2 mds.0.23 boot_start 1: opening snap table
-86> 2013-03-13 11:07:32.845296 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/26568, 111: Connection refused
-85> 2013-03-13 11:07:32.845354 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6807/26872 -- osd_op(mds.0.23:4 mds_snaptable [read 0~0] 1.d90270ad e586) v4 -- ?+0 0x1cd3d80 con 0x1ceab00
-84> 2013-03-13 11:07:32.845356 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-83> 2013-03-13 11:07:32.845368 7fe08b673700 2 mds.0.23 boot_start 1: opening mds log
-82> 2013-03-13 11:07:32.845384 7fe08b673700 5 mds.0.log open discovering log bounds
-81> 2013-03-13 11:07:32.845385 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault
-80> 2013-03-13 11:07:32.845392 7fe08b673700 1 mds.0.journaler(ro) recover start
-79> 2013-03-13 11:07:32.845394 7fe08b673700 1 mds.0.journaler(ro) read_head
-78> 2013-03-13 11:07:32.845407 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6802/26568 -- osd_op(mds.0.23:5 200.00000000 [read 0~0] 1.844f3494 e586) v4 -- ?+0 0x1cd3b40 con 0x1cea840
-77> 2013-03-13 11:07:32.845411 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6802/26568, 111: Connection refused
-76> 2013-03-13 11:07:32.845422 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-75> 2013-03-13 11:07:32.845422 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 11 ==== osd_map(586..586 src has 85..586) v3 ==== 3230+0+0 (3173237601 0 0) 0x1cff200 con 0x1cea580
-74> 2013-03-13 11:07:32.845397 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-73> 2013-03-13 11:07:32.845434 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x1cee700 con 0x1cea580
-72> 2013-03-13 11:07:32.845440 7fe08b673700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
-71> 2013-03-13 11:07:32.845441 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-70> 2013-03-13 11:07:32.845450 7fe088c6c700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault
-69> 2013-03-13 11:07:32.845472 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-68> 2013-03-13 11:07:32.845481 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-67> 2013-03-13 11:07:33.045611 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-66> 2013-03-13 11:07:33.045651 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-65> 2013-03-13 11:07:33.445828 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-64> 2013-03-13 11:07:33.445867 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-63> 2013-03-13 11:07:34.246035 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-62> 2013-03-13 11:07:34.246070 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-61> 2013-03-13 11:07:35.482245 7fe08a671700 10 monclient: tick
-60> 2013-03-13 11:07:35.482282 7fe08a671700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 11:07:05.482280)
-59> 2013-03-13 11:07:35.482302 7fe08a671700 10 monclient: renew subs? (now: 2013-03-13 11:07:35.482301; renew after: 2013-03-13 11:10:02.484278) -- no
-58> 2013-03-13 11:07:35.846216 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-57> 2013-03-13 11:07:35.846255 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-56> 2013-03-13 11:07:36.484360 7fe08956e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-55> 2013-03-13 11:07:36.484393 7fe08956e700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mdsbeacon(16098/a up:replay seq 2 v1219) v2 -- ?+0 0x1cf72c0 con 0x1cea580
-54> 2013-03-13 11:07:36.484772 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 13 ==== mdsbeacon(16098/a up:replay seq 2 v1219) v2 ==== 103+0+0 (3858646558 0 0) 0x1cf7000 con 0x1cea580
-53> 2013-03-13 11:07:37.331995 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56092 s=1 pgs=0 cs=0 l=1).connect claims to be 0.0.0.0:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
-52> 2013-03-13 11:07:37.332031 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56092 s=1 pgs=0 cs=0 l=1).fault 107: Transport endpoint is not connected
-51> 2013-03-13 11:07:37.732394 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56106 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
-50> 2013-03-13 11:07:37.732428 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56106 s=1 pgs=0 cs=0 l=1).fault 107: Transport endpoint is not connected
-49> 2013-03-13 11:07:38.532740 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56109 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
-48> 2013-03-13 11:07:38.532775 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56109 s=1 pgs=0 cs=0 l=1).fault 107: Transport endpoint is not connected
-47> 2013-03-13 11:07:39.046391 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-46> 2013-03-13 11:07:39.046426 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-45> 2013-03-13 11:07:40.133049 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56116 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
-44> 2013-03-13 11:07:40.133079 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56116 s=1 pgs=0 cs=0 l=1).fault 107: Transport endpoint is not connected
-43> 2013-03-13 11:07:40.484543 7fe08956e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-42> 2013-03-13 11:07:40.484563 7fe08956e700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mdsbeacon(16098/a up:replay seq 3 v1219) v2 -- ?+0 0x1cf7580 con 0x1cea580
-41> 2013-03-13 11:07:40.484888 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 14 ==== mdsbeacon(16098/a up:replay seq 3 v1219) v2 ==== 103+0+0 (2631351161 0 0) 0x1cf7b00 con 0x1cea580
-40> 2013-03-13 11:07:43.369741 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56122 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
-39> 2013-03-13 11:07:43.369775 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56122 s=1 pgs=0 cs=0 l=1).fault 107: Transport endpoint is not connected
-38> 2013-03-13 11:07:44.484736 7fe08956e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-37> 2013-03-13 11:07:44.484759 7fe08956e700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mdsbeacon(16098/a up:replay seq 4 v1219) v2 -- ?+0 0x1cf72c0 con 0x1cea580
-36> 2013-03-13 11:07:44.799182 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 15 ==== mdsbeacon(16098/a up:replay seq 4 v1219) v2 ==== 103+0+0 (4142219197 0 0) 0x1cf7840 con 0x1cea580
-35> 2013-03-13 11:07:45.446605 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-34> 2013-03-13 11:07:45.446643 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=18 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-33> 2013-03-13 11:07:45.482392 7fe08a671700 10 monclient: tick
-32> 2013-03-13 11:07:45.482420 7fe08a671700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-03-13 11:07:15.482418)
-31> 2013-03-13 11:07:45.482433 7fe08a671700 10 monclient: renew subs? (now: 2013-03-13 11:07:45.482432; renew after: 2013-03-13 11:10:02.484278) -- no
-30> 2013-03-13 11:07:47.484500 7fe08956e700 10 monclient: renew_subs
-29> 2013-03-13 11:07:47.484515 7fe08956e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-28> 2013-03-13 11:07:47.484528 7fe08956e700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mon_subscribe({mdsmap=1220+,monmap=2+,osdmap=587}) v2 -- ?+0 0x1ceea80 con 0x1cea580
-27> 2013-03-13 11:07:47.484550 7fe08956e700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6807/26872 -- ping v1 -- ?+0 0x1cee540 con 0x1ceab00
-26> 2013-03-13 11:07:47.484571 7fe08956e700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6802/26568 -- ping v1 -- ?+0 0x1d0f380 con 0x1cea840
-25> 2013-03-13 11:07:47.484655 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=19 :0 s=1 pgs=0 cs=0 l=1).connect error 10.214.136.10:6807/26872, 111: Connection refused
-24> 2013-03-13 11:07:47.484698 7fe088c6c700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6807/26872 pipe(0x1d01280 sd=19 :0 s=1 pgs=0 cs=0 l=1).fault 111: Connection refused
-23> 2013-03-13 11:07:47.484833 7fe088d6d700 0 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56129 s=1 pgs=0 cs=0 l=1).connect claims to be 10.214.136.10:6802/40956 not 10.214.136.10:6802/26568 - wrong node!
-22> 2013-03-13 11:07:47.484893 7fe088d6d700 2 -- 10.214.136.10:6800/40709 >> 10.214.136.10:6802/26568 pipe(0x1d01500 sd=17 :56129 s=1 pgs=0 cs=0 l=1).fault 107: Transport endpoint is not connected
-21> 2013-03-13 11:07:47.485099 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 16 ==== osd_map(587..589 src has 88..589) v3 ==== 1370+0+0 (3867779569 0 0) 0x1cff000 con 0x1cea580
-20> 2013-03-13 11:07:47.485164 7fe08b673700 1 -- 10.214.136.10:6800/40709 mark_down 0x1cea840 -- 0x1d01500
-19> 2013-03-13 11:07:47.485176 7fe08b673700 1 -- 10.214.136.10:6800/40709 mark_down 0x1ceab00 -- 0x1d01280
-18> 2013-03-13 11:07:47.485402 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6801/40956 -- osd_op(mds.0.23:1 mds0_inotable [read 0~0] 1.b852b893 RETRY=1 e589) v4 -- ?+0 0x1cd3480 con 0x1cea9a0
-17> 2013-03-13 11:07:47.485425 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6801/40956 -- osd_op(mds.0.23:2 mds0_sessionmap [read 0~0] 1.3270c60b RETRY=1 e589) v4 -- ?+0 0x1cd3900 con 0x1cea9a0
-16> 2013-03-13 11:07:47.485433 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6801/40956 -- osd_op(mds.0.23:3 mds_anchortable [read 0~0] 1.a977f6a7 RETRY=1 e589) v4 -- ?+0 0x1cd36c0 con 0x1cea9a0
-15> 2013-03-13 11:07:47.485440 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6804/41191 -- osd_op(mds.0.23:4 mds_snaptable [read 0~0] 1.d90270ad RETRY=1 e589) v4 -- ?+0 0x1cd3b40 con 0x1ceadc0
-14> 2013-03-13 11:07:47.485448 7fe08b673700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6801/40956 -- osd_op(mds.0.23:5 200.00000000 [read 0~0] 1.844f3494 RETRY=1 e589) v4 -- ?+0 0x1d08b40 con 0x1cea9a0
-13> 2013-03-13 11:07:47.485469 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 17 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1455940820 0 0) 0x1d0f8c0 con 0x1cea580
-12> 2013-03-13 11:07:47.485477 7fe08b673700 10 monclient: handle_subscribe_ack sent 2013-03-13 11:07:47.484512 renew after 2013-03-13 11:10:17.484512
-11> 2013-03-13 11:07:47.485704 7fe08b673700 0 mds.0.23 ms_handle_connect on 10.214.136.10:6801/40956
-10> 2013-03-13 11:07:47.486085 7fe08b673700 0 mds.0.23 ms_handle_connect on 10.214.136.10:6804/41191
-9> 2013-03-13 11:07:47.666416 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== osd.1 10.214.136.10:6804/41191 1 ==== osd_op_reply(4 mds_snaptable [read 0~41] ondisk = 0) v4 ==== 112+0+41 (2282816424 0 2312406677) 0x1cda400 con 0x1ceadc0
-8> 2013-03-13 11:07:48.484855 7fe08956e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-7> 2013-03-13 11:07:48.484878 7fe08956e700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mdsbeacon(16098/a up:replay seq 5 v1219) v2 -- ?+0 0x1cf7dc0 con 0x1cea580
-6> 2013-03-13 11:07:48.567933 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 18 ==== mdsbeacon(16098/a up:replay seq 5 v1219) v2 ==== 103+0+0 (2412528346 0 0) 0x1cf7dc0 con 0x1cea580
-5> 2013-03-13 11:07:52.405309 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== osd.0 10.214.136.10:6801/40956 1 ==== osd_op_reply(1 mds0_inotable [read 0~29] ondisk = 0) v4 ==== 112+0+29 (3341700367 0 2795693049) 0x1cff600 con 0x1cea9a0
-4> 2013-03-13 11:07:52.484983 7fe08956e700 10 monclient: _send_mon_message to mon.a at 10.214.136.10:6789/0
-3> 2013-03-13 11:07:52.485006 7fe08956e700 1 -- 10.214.136.10:6800/40709 --> 10.214.136.10:6789/0 -- mdsbeacon(16098/a up:replay seq 6 v1219) v2 -- ?+0 0x1cf7b00 con 0x1cea580
-2> 2013-03-13 11:07:52.485392 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== mon.0 10.214.136.10:6789/0 19 ==== mdsbeacon(16098/a up:replay seq 6 v1219) v2 ==== 103+0+0 (79152499 0 0) 0x1cf72c0 con 0x1cea580
-1> 2013-03-13 11:07:53.919457 7fe08b673700 1 -- 10.214.136.10:6800/40709 <== osd.0 10.214.136.10:6801/40956 2 ==== osd_op_reply(2 mds0_sessionmap [read 0~1848] ondisk = 0) v4 ==== 114+0+1848 (2411378392 0 1211354084) 0x1cda400 con 0x1cea9a0
0> 2013-03-13 11:07:53.921086 7fe08b673700 -1 *** Caught signal (Aborted) **
in thread 7fe08b673700

ceph version 0.58-501-g66be33a (66be33a3e783b605e46d735a008d6ba79f58a63d)
1: /usr/bin/ceph-mds() [0x84fa7a]
2: (()+0xfcb0) [0x7fe08f9d7cb0]
3: (gsignal()+0x35) [0x7fe08e0c8425]
4: (abort()+0x17b) [0x7fe08e0cbb8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fe08ea1a69d]
6: (()+0xb5846) [0x7fe08ea18846]
7: (()+0xb5873) [0x7fe08ea18873]
8: (()+0xb596e) [0x7fe08ea1896e]
9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x127) [0x7adcd7]
10: (void decode<unsigned long, inodeno_t>(std::map<unsigned long, inodeno_t, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, inodeno_t> > >&, ceph::buffer::list::iterator&)+0xde) [0x6b2cae]
11: (session_info_t::decode(ceph::buffer::list::iterator&)+0x139) [0x7d7ee9]
12: (SessionMap::decode(ceph::buffer::list::iterator&)+0xb98) [0x6c87e8]
13: (SessionMap::_load_finish(int, ceph::buffer::list&)+0x83) [0x6c8cc3]
14: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0xe38) [0x6f17b8]
15: (MDS::handle_core_message(Message*)+0xae8) [0x4db088]
16: (MDS::_dispatch(Message*)+0x2f) [0x4db24f]
17: (MDS::ms_dispatch(Message*)+0x1db) [0x4dcceb]
18: (DispatchQueue::entry()+0x341) [0x81e1a1]
19: (DispatchQueue::DispatchThread::entry()+0xd) [0x79c45d]
20: (()+0x7e9a) [0x7fe08f9cfe9a]
21: (clone()+0x6d) [0x7fe08e185cbd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-mds.a.log
--- end dump of recent events ---
    (1-1/1)