016-08-24 11:28:45.027028 7fb261aba700 -1 mds/journal.cc: In function 'virtual void EImportStart::replay(MDSRank*)' thread 7fb261aba700 time 2016-08-24 11:28:45.025881 mds/journal.cc: 2929: FAILED assert(mds->sessionmap.get_version() == cmapv) ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55f98ea972a0] 2: (EImportStart::replay(MDSRank*)+0x9e5) [0x55f98e970e75] 3: (MDLog::_replay_thread()+0xe73) [0x55f98e8efb23] 4: (MDLog::ReplayThread::entry()+0xd) [0x55f98e69a84d] 5: (()+0x75ca) [0x7fb26dab85ca] 6: (clone()+0x6d) [0x7fb26c4f8f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -233> 2016-08-24 11:28:35.665027 7fb26e6a9200 5 asok(0x55f999934000) register_command perfcounters_dump hook 0x55f999918030 -232> 2016-08-24 11:28:35.665046 7fb26e6a9200 5 asok(0x55f999934000) register_command 1 hook 0x55f999918030 -231> 2016-08-24 11:28:35.665051 7fb26e6a9200 5 asok(0x55f999934000) register_command perf dump hook 0x55f999918030 -230> 2016-08-24 11:28:35.665056 7fb26e6a9200 5 asok(0x55f999934000) register_command perfcounters_schema hook 0x55f999918030 -229> 2016-08-24 11:28:35.665060 7fb26e6a9200 5 asok(0x55f999934000) register_command 2 hook 0x55f999918030 -228> 2016-08-24 11:28:35.665064 7fb26e6a9200 5 asok(0x55f999934000) register_command perf schema hook 0x55f999918030 -227> 2016-08-24 11:28:35.665069 7fb26e6a9200 5 asok(0x55f999934000) register_command perf reset hook 0x55f999918030 -226> 2016-08-24 11:28:35.665074 7fb26e6a9200 5 asok(0x55f999934000) register_command config show hook 0x55f999918030 -225> 2016-08-24 11:28:35.665078 7fb26e6a9200 5 asok(0x55f999934000) register_command config set hook 0x55f999918030 -224> 2016-08-24 11:28:35.665083 7fb26e6a9200 5 asok(0x55f999934000) register_command config get hook 0x55f999918030 -223> 2016-08-24 11:28:35.665088 7fb26e6a9200 5 asok(0x55f999934000) register_command config diff hook 0x55f999918030 -222> 2016-08-24 11:28:35.665092 7fb26e6a9200 5 asok(0x55f999934000) register_command log flush hook 0x55f999918030 -221> 2016-08-24 11:28:35.665097 7fb26e6a9200 5 asok(0x55f999934000) register_command log dump hook 0x55f999918030 -220> 2016-08-24 11:28:35.665101 7fb26e6a9200 5 asok(0x55f999934000) register_command log reopen hook 0x55f999918030 -219> 2016-08-24 11:28:35.670400 7fb26e6a9200 0 set uid:gid to 167:167 (ceph:ceph) -218> 2016-08-24 11:28:35.670448 7fb26e6a9200 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mds, pid 22354 -217> 2016-08-24 11:28:35.670693 7fb26e6a9200 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/22354 need_addr=1 -216> 2016-08-24 11:28:35.670706 7fb26e6a9200 0 pidfile_write: ignore empty --pid-file -215> 2016-08-24 11:28:35.672654 7fb26e6a9200 5 asok(0x55f999934000) init /var/run/ceph/ceph-mds.switcheroo.asok -214> 2016-08-24 11:28:35.672678 7fb26e6a9200 5 asok(0x55f999934000) bind_and_listen /var/run/ceph/ceph-mds.switcheroo.asok -213> 2016-08-24 11:28:35.672931 7fb26e6a9200 5 asok(0x55f999934000) register_command 0 hook 0x55f9999160a8 -212> 2016-08-24 11:28:35.672947 7fb26e6a9200 5 asok(0x55f999934000) register_command version hook 0x55f9999160a8 -211> 2016-08-24 11:28:35.672958 7fb26e6a9200 5 asok(0x55f999934000) register_command git_version hook 0x55f9999160a8 -210> 2016-08-24 11:28:35.672970 7fb26e6a9200 5 asok(0x55f999934000) register_command help hook 0x55f999918130 -209> 2016-08-24 11:28:35.672981 7fb26e6a9200 5 asok(0x55f999934000) register_command get_command_descriptions hook 0x55f999918140 -208> 2016-08-24 11:28:35.673026 7fb269dd4700 5 asok(0x55f999934000) entry start -207> 2016-08-24 11:28:35.673032 7fb26e6a9200 10 monclient(hunting): build_initial_monmap -206> 2016-08-24 11:28:35.673064 7fb26e6a9200 1 -- 0.0.0.0:6800/22354 messenger.start -205> 2016-08-24 11:28:35.673107 7fb26e6a9200 5 adding auth protocol: cephx -204> 2016-08-24 11:28:35.673119 7fb26e6a9200 5 adding auth protocol: cephx -203> 2016-08-24 11:28:35.673184 7fb26e6a9200 5 asok(0x55f999934000) register_command objecter_requests hook 0x55f999918150 -202> 2016-08-24 11:28:35.673271 7fb26e6a9200 1 accepter.accepter.start -201> 2016-08-24 11:28:35.673304 7fb26e6a9200 10 monclient(hunting): init -200> 2016-08-24 11:28:35.673316 7fb26e6a9200 5 adding auth protocol: cephx -199> 2016-08-24 11:28:35.673325 7fb26e6a9200 10 monclient(hunting): auth_supported 2 method cephx -198> 2016-08-24 11:28:35.673503 7fb26e6a9200 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-switcheroo/keyring -197> 2016-08-24 11:28:35.673566 7fb26e6a9200 10 monclient(hunting): _reopen_session rank -1 name -196> 2016-08-24 11:28:35.673710 7fb26e6a9200 10 monclient(hunting): picked mon.noname-a con 0x55f999968300 addr 3.193.148.63:6789/0 -195> 2016-08-24 11:28:35.676253 7fb26e6a9200 10 monclient(hunting): _send_mon_message to mon.noname-a at 3.193.148.63:6789/0 -194> 2016-08-24 11:28:35.676259 7fb26e6a9200 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- auth(proto 0 35 bytes epoch 0) v1 -- ?+0 0x55f99994ca00 con 0x55f999968300 -193> 2016-08-24 11:28:35.676267 7fb26e6a9200 10 monclient(hunting): renew_subs -192> 2016-08-24 11:28:35.675576 7fb26e6a3700 1 -- 3.193.148.63:6800/22354 learned my addr 3.193.148.63:6800/22354 -191> 2016-08-24 11:28:35.677630 7fb265dcc700 2 -- 3.193.148.63:6800/22354 >> 3.193.148.63:6789/0 pipe(0x55f9999fe000 sd=8 :33246 s=2 pgs=6959240 cs=1 l=1 c=0x55f999968300).reader got KEEPALIVE_ACK -190> 2016-08-24 11:28:35.680248 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 1 ==== mon_map magic: 0 v1 ==== 503+0+0 (716945685 0 0) 0x55f999930900 con 0x55f999968300 -189> 2016-08-24 11:28:35.680261 7fb2685d1700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -188> 2016-08-24 11:28:35.680270 7fb2685d1700 10 monclient(hunting): got monmap 5, mon.noname-a is now rank -1 -187> 2016-08-24 11:28:35.680272 7fb2685d1700 10 monclient(hunting): dump: epoch 5 fsid 40216c8a-ba3c-4cec-9e80-212396e214a1 last_changed 2016-05-05 10:18:29.089450 created 0.000000 0: 3.193.148.63:6789/0 mon.switcheroo 1: 3.193.149.245:6789/0 mon.microstation 2: 3.193.150.32:6789/0 mon.dashboardpc -186> 2016-08-24 11:28:35.680293 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3800149130 0 0) 0x55f99994d180 con 0x55f999968300 -185> 2016-08-24 11:28:35.680309 7fb2685d1700 10 monclient(hunting): my global_id is 24534054 -184> 2016-08-24 11:28:35.680387 7fb2685d1700 10 monclient(hunting): _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -183> 2016-08-24 11:28:35.680391 7fb2685d1700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x55f99994cf00 con 0x55f999968300 -182> 2016-08-24 11:28:35.682239 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (1427904375 0 0) 0x55f99994d400 con 0x55f999968300 -181> 2016-08-24 11:28:35.682311 7fb2685d1700 10 monclient(hunting): _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -180> 2016-08-24 11:28:35.682315 7fb2685d1700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0 0x55f99994d180 con 0x55f999968300 -179> 2016-08-24 11:28:35.684237 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (1497174212 0 0) 0x55f99994ca00 con 0x55f999968300 -178> 2016-08-24 11:28:35.684344 7fb2685d1700 1 monclient(hunting): found mon.switcheroo -177> 2016-08-24 11:28:35.684346 7fb2685d1700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -176> 2016-08-24 11:28:35.684349 7fb2685d1700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x55f999925000 con 0x55f999968300 -175> 2016-08-24 11:28:35.684363 7fb2685d1700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2016-08-24 11:28:05.684362) -174> 2016-08-24 11:28:35.684369 7fb2685d1700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -173> 2016-08-24 11:28:35.684373 7fb2685d1700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x55f99994d400 con 0x55f999968300 -172> 2016-08-24 11:28:35.684390 7fb26e6a9200 5 monclient: authenticate success, global_id 24534054 -171> 2016-08-24 11:28:35.684402 7fb26e6a9200 10 monclient: wait_auth_rotating waiting (until 2016-08-24 11:29:05.684402) -170> 2016-08-24 11:28:35.745238 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 503+0+0 (716945685 0 0) 0x55f999930b40 con 0x55f999968300 -169> 2016-08-24 11:28:35.745258 7fb2685d1700 10 monclient: handle_monmap mon_map magic: 0 v1 -168> 2016-08-24 11:28:35.745268 7fb2685d1700 10 monclient: got monmap 5, mon.switcheroo is now rank 0 -167> 2016-08-24 11:28:35.745270 7fb2685d1700 10 monclient: dump: epoch 5 fsid 40216c8a-ba3c-4cec-9e80-212396e214a1 last_changed 2016-05-05 10:18:29.089450 created 0.000000 0: 3.193.148.63:6789/0 mon.switcheroo 1: 3.193.149.245:6789/0 mon.microstation 2: 3.193.150.32:6789/0 mon.dashboardpc -166> 2016-08-24 11:28:35.745287 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (3799675998 0 0) 0x55f99994d680 con 0x55f999968300 -165> 2016-08-24 11:28:35.745393 7fb2685d1700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2016-08-24 11:28:05.745392) -164> 2016-08-24 11:28:35.745413 7fb26e6a9200 10 monclient: wait_auth_rotating done -163> 2016-08-24 11:28:35.745463 7fb26e6a9200 10 monclient: renew_subs -162> 2016-08-24 11:28:35.745474 7fb26e6a9200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -161> 2016-08-24 11:28:35.745486 7fb26e6a9200 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x55f999925200 con 0x55f999968300 -160> 2016-08-24 11:28:35.745506 7fb26e6a9200 10 monclient: renew_subs -159> 2016-08-24 11:28:35.745516 7fb26e6a9200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -158> 2016-08-24 11:28:35.745527 7fb26e6a9200 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mon_subscribe({mdsmap=0+}) v2 -- ?+0 0x55f999925400 con 0x55f999968300 -157> 2016-08-24 11:28:35.745967 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 7 ==== osd_map(632275..632275 src has 631672..632275) v3 ==== 32550+0+0 (1659082127 0 0) 0x55f99994ca00 con 0x55f999968300 -156> 2016-08-24 11:28:35.746172 7fb26e6a9200 5 asok(0x55f999934000) register_command status hook 0x55f999918190 -155> 2016-08-24 11:28:35.746189 7fb26e6a9200 5 asok(0x55f999934000) register_command dump_ops_in_flight hook 0x55f999918190 -154> 2016-08-24 11:28:35.746191 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 8 ==== mdsmap(e 1138272) v1 ==== 462+0+0 (3514661578 0 0) 0x55f999931680 con 0x55f999968300 -153> 2016-08-24 11:28:35.746204 7fb2685d1700 5 mds.switcheroo handle_mds_map epoch 1138272 from mon.0 -152> 2016-08-24 11:28:35.746205 7fb26e6a9200 5 asok(0x55f999934000) register_command ops hook 0x55f999918190 -151> 2016-08-24 11:28:35.746222 7fb26e6a9200 5 asok(0x55f999934000) register_command dump_blocked_ops hook 0x55f999918190 -150> 2016-08-24 11:28:35.746241 7fb26e6a9200 5 asok(0x55f999934000) register_command dump_historic_ops hook 0x55f999918190 -149> 2016-08-24 11:28:35.746255 7fb26e6a9200 5 asok(0x55f999934000) register_command scrub_path hook 0x55f999918190 -148> 2016-08-24 11:28:35.746266 7fb26e6a9200 5 asok(0x55f999934000) register_command tag path hook 0x55f999918190 -147> 2016-08-24 11:28:35.746277 7fb26e6a9200 5 asok(0x55f999934000) register_command flush_path hook 0x55f999918190 -146> 2016-08-24 11:28:35.746288 7fb26e6a9200 5 asok(0x55f999934000) register_command export dir hook 0x55f999918190 -145> 2016-08-24 11:28:35.746299 7fb26e6a9200 5 asok(0x55f999934000) register_command dump cache hook 0x55f999918190 -144> 2016-08-24 11:28:35.746311 7fb26e6a9200 5 asok(0x55f999934000) register_command session evict hook 0x55f999918190 -143> 2016-08-24 11:28:35.746322 7fb26e6a9200 5 asok(0x55f999934000) register_command osdmap barrier hook 0x55f999918190 -142> 2016-08-24 11:28:35.746334 7fb26e6a9200 5 asok(0x55f999934000) register_command session ls hook 0x55f999918190 -141> 2016-08-24 11:28:35.746346 7fb26e6a9200 5 asok(0x55f999934000) register_command flush journal hook 0x55f999918190 -140> 2016-08-24 11:28:35.746357 7fb26e6a9200 5 asok(0x55f999934000) register_command force_readonly hook 0x55f999918190 -139> 2016-08-24 11:28:35.746367 7fb26e6a9200 5 asok(0x55f999934000) register_command get subtrees hook 0x55f999918190 -138> 2016-08-24 11:28:35.746380 7fb26e6a9200 5 asok(0x55f999934000) register_command dirfrag split hook 0x55f999918190 -137> 2016-08-24 11:28:35.746391 7fb26e6a9200 5 asok(0x55f999934000) register_command dirfrag merge hook 0x55f999918190 -136> 2016-08-24 11:28:35.746402 7fb26e6a9200 5 asok(0x55f999934000) register_command dirfrag ls hook 0x55f999918190 -135> 2016-08-24 11:28:35.802355 7fb26e6a9200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -134> 2016-08-24 11:28:35.802378 7fb26e6a9200 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mdsbeacon(24534054/switcheroo up:boot seq 1 v1138272) v7 -- ?+0 0x55f999a28000 con 0x55f999968300 -133> 2016-08-24 11:28:36.885446 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 9 ==== mdsmap(e 1138273) v1 ==== 687+0+0 (3096424815 0 0) 0x55f9999318c0 con 0x55f999968300 -132> 2016-08-24 11:28:36.885471 7fb2685d1700 5 mds.switcheroo handle_mds_map epoch 1138273 from mon.0 -131> 2016-08-24 11:28:36.885521 7fb2685d1700 1 mds.switcheroo handle_mds_map standby -130> 2016-08-24 11:28:36.885532 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 10 ==== mdsbeacon(24534054/switcheroo up:boot seq 1 v1138273) v7 ==== 139+0+0 (393677139 0 0) 0x55f999a28380 con 0x55f999968300 -129> 2016-08-24 11:28:36.976901 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 11 ==== mdsmap(e 1138274) v1 ==== 968+0+0 (2912268178 0 0) 0x55f999931b00 con 0x55f999968300 -128> 2016-08-24 11:28:36.976921 7fb2685d1700 5 mds.switcheroo handle_mds_map epoch 1138274 from mon.0 -127> 2016-08-24 11:28:36.977034 7fb2685d1700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -126> 2016-08-24 11:28:36.977074 7fb2685d1700 10 monclient: renew_subs -125> 2016-08-24 11:28:36.977082 7fb2685d1700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -124> 2016-08-24 11:28:36.977092 7fb2685d1700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=632276}) v2 -- ?+0 0x55f999925000 con 0x55f999968300 -123> 2016-08-24 11:28:36.977191 7fb2685d1700 1 mds.1.1138274 handle_mds_map i am now mds.1.1138274 -122> 2016-08-24 11:28:36.977196 7fb2685d1700 1 mds.1.1138274 handle_mds_map state change up:boot --> up:replay -121> 2016-08-24 11:28:36.977206 7fb2685d1700 1 mds.1.1138274 replay_start -120> 2016-08-24 11:28:36.977210 7fb2685d1700 1 mds.1.1138274 recovery set is 0 -119> 2016-08-24 11:28:36.977217 7fb2685d1700 1 mds.1.1138274 waiting for osdmap 632276 (which blacklists prior instance) -118> 2016-08-24 11:28:37.009508 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 12 ==== osd_map(632276..632276 src has 631672..632276) v3 ==== 358+0+0 (1016501818 0 0) 0x55f99994cf00 con 0x55f999968300 -117> 2016-08-24 11:28:37.009573 7fb263cc7700 2 mds.1.1138274 boot_start 0: opening inotable -116> 2016-08-24 11:28:37.009703 7fb263cc7700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138274:1 6.ad6e75db mds1_inotable [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1c680 con 0x55f999968f00 -115> 2016-08-24 11:28:37.009735 7fb263cc7700 2 mds.1.1138274 boot_start 0: opening sessionmap -114> 2016-08-24 11:28:37.009797 7fb263cc7700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.154:6812/2220 -- osd_op(mds.1.1138274:2 6.32a2fe43 mds1_sessionmap [omap-get-header 0~0,omap-get-vals 0~16] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1c9c0 con 0x55f999969080 -113> 2016-08-24 11:28:37.009820 7fb263cc7700 2 mds.1.1138274 boot_start 0: opening mds log -112> 2016-08-24 11:28:37.009841 7fb263cc7700 5 mds.1.log open discovering log bounds -111> 2016-08-24 11:28:37.009923 7fb2632c4700 4 mds.1.journalpointer Reading journal pointer '401.00000000' -110> 2016-08-24 11:28:37.009988 7fb2632c4700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.154:6800/1414 -- osd_op(mds.1.1138274:3 6.fcd6c765 401.00000000 [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1d040 con 0x55f999969380 -109> 2016-08-24 11:28:37.011093 7fb2685d1700 10 monclient: renew_subs -108> 2016-08-24 11:28:37.011101 7fb2685d1700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -107> 2016-08-24 11:28:37.011105 7fb2685d1700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=632277}) v2 -- ?+0 0x55f999926400 con 0x55f999968300 -106> 2016-08-24 11:28:37.012542 7fb2621c1700 1 -- 3.193.148.63:6800/22354 <== osd.3 3.193.149.245:6800/7279 1 ==== osd_op_reply(1 mds1_inotable [read 0~98] v0'0 uv3885 ondisk = 0) v7 ==== 133+0+98 (4042972645 0 3676677995) 0x55f999a1d380 con 0x55f999968f00 -105> 2016-08-24 11:28:37.013344 7fb2620c0700 1 -- 3.193.148.63:6800/22354 <== osd.8 3.193.149.154:6800/1414 1 ==== osd_op_reply(3 401.00000000 [read 0~22] v0'0 uv1 ondisk = 0) v7 ==== 132+0+22 (398738691 0 369460326) 0x55f999a1d6c0 con 0x55f999969380 -104> 2016-08-24 11:28:37.013374 7fb2632c4700 1 mds.1.journaler(ro) recover start -103> 2016-08-24 11:28:37.013377 7fb2632c4700 1 mds.1.journaler(ro) read_head -102> 2016-08-24 11:28:37.013399 7fb2632c4700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.154:6812/2220 -- osd_op(mds.1.1138274:4 6.5664a157 201.00000000 [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1da00 con 0x55f999969080 -101> 2016-08-24 11:28:37.013413 7fb2632c4700 4 mds.1.log Waiting for journal 201 to recover... -100> 2016-08-24 11:28:37.013719 7fb261fbf700 1 -- 3.193.148.63:6800/22354 <== osd.6 3.193.149.154:6812/2220 1 ==== osd_op_reply(2 mds1_sessionmap [omap-get-header 0~0,omap-get-vals 0~16] v0'0 uv2124 ondisk = 0) v7 ==== 177+0+807 (906194814 0 2854467212) 0x55f999a1c9c0 con 0x55f999969080 -99> 2016-08-24 11:28:37.014370 7fb261fbf700 1 -- 3.193.148.63:6800/22354 <== osd.6 3.193.149.154:6812/2220 2 ==== osd_op_reply(4 201.00000000 [read 0~90] v0'0 uv20080 ondisk = 0) v7 ==== 132+0+90 (3545930061 0 3155850792) 0x55f999a1c9c0 con 0x55f999969080 -98> 2016-08-24 11:28:37.014396 7fb263cc7700 1 mds.1.journaler(ro) _finish_read_head loghead(trim 838860800, expire 841892677, write 890374656, stream_format 1). probing for end of log (from 890374656)... -97> 2016-08-24 11:28:37.014399 7fb263cc7700 1 mds.1.journaler(ro) probing for end of the log -96> 2016-08-24 11:28:37.014477 7fb263cc7700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6801/1173 -- osd_op(mds.1.1138274:5 6.b16d511 201.000000d4 [stat] snapc 0=[] ack+read+rwordered+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1cd00 con 0x55f999969200 -95> 2016-08-24 11:28:37.014532 7fb263cc7700 1 -- 3.193.148.63:6800/22354 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138274:6 6.3071b1a0 201.000000d5 [stat] snapc 0=[] ack+read+rwordered+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1da00 con 0x55f999969b00 -94> 2016-08-24 11:28:37.015672 7fb261cbc700 1 -- 3.193.148.63:6800/22354 <== osd.0 3.193.148.63:6801/1173 1 ==== osd_op_reply(5 201.000000d4 [stat] v0'0 uv2195 ondisk = 0) v7 ==== 132+0+16 (2266058301 0 2553466074) 0x55f999a1e700 con 0x55f999969200 -93> 2016-08-24 11:28:37.016155 7fb261bbb700 1 -- 3.193.148.63:6800/22354 <== osd.2 3.193.150.32:6800/23404 1 ==== osd_op_reply(6 201.000000d5 [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 (521958358 0 0) 0x55f999a1ea40 con 0x55f999969b00 -92> 2016-08-24 11:28:37.016196 7fb263cc7700 1 mds.1.journaler(ro) _finish_probe_end write_pos = 890845128 (header had 890374656). recovered. -91> 2016-08-24 11:28:37.016207 7fb2632c4700 4 mds.1.log Journal 201 recovered. -90> 2016-08-24 11:28:37.016209 7fb2632c4700 4 mds.1.log Recovered journal 201 in format 1 -89> 2016-08-24 11:28:37.016214 7fb2632c4700 2 mds.1.1138274 boot_start 1: loading/discovering base inodes -88> 2016-08-24 11:28:37.016224 7fb2632c4700 0 mds.1.cache creating system inode with ino:101 -87> 2016-08-24 11:28:37.016326 7fb2632c4700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138274:7 6.75b54883 101.00000000 [getxattr inode] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1dd40 con 0x55f999968f00 -86> 2016-08-24 11:28:37.016363 7fb2632c4700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138274:8 6.2fd1f055 101.00000000.inode [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1ed80 con 0x55f999968f00 -85> 2016-08-24 11:28:37.016374 7fb2632c4700 0 mds.1.cache creating system inode with ino:1 -84> 2016-08-24 11:28:37.017234 7fb2621c1700 1 -- 3.193.148.63:6800/22354 <== osd.3 3.193.149.245:6800/7279 2 ==== osd_op_reply(8 101.00000000.inode [read 0~508] v0'0 uv42 ondisk = 0) v7 ==== 138+0+508 (3934538994 0 1447057696) 0x55f999a1d380 con 0x55f999968f00 -83> 2016-08-24 11:28:37.017332 7fb2621c1700 1 -- 3.193.148.63:6800/22354 <== osd.3 3.193.149.245:6800/7279 3 ==== osd_op_reply(7 101.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v7 ==== 132+0+0 (2028968740 0 0) 0x55f999a1d380 con 0x55f999968f00 -82> 2016-08-24 11:28:37.017370 7fb263cc7700 2 mds.1.1138274 boot_start 2: replaying mds log -81> 2016-08-24 11:28:37.017445 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138274:9 6.8aa3d078 201.000000c8 [read 3031877~1162427 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1c680 con 0x55f999968f00 -80> 2016-08-24 11:28:37.017517 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.154:6800/1414 -- osd_op(mds.1.1138274:10 6.fba23765 201.000000c9 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1dd40 con 0x55f999969380 -79> 2016-08-24 11:28:37.017588 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138274:11 6.25f51ba0 201.000000ca [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1f400 con 0x55f999969b00 -78> 2016-08-24 11:28:37.017713 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.154:6804/1537 -- osd_op(mds.1.1138274:12 6.aafb22f0 201.000000cb [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1da00 con 0x55f99996a280 -77> 2016-08-24 11:28:37.017739 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138274:13 6.4dc0c2b3 201.000000cc [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1f400 con 0x55f999969b00 -76> 2016-08-24 11:28:37.017842 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138274:14 6.587f473b 201.000000cd [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1d040 con 0x55f99996a400 -75> 2016-08-24 11:28:37.017873 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138274:15 6.531b5b90 201.000000ce [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1dd40 con 0x55f99996a400 -74> 2016-08-24 11:28:37.017932 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138274:16 6.766afc26 201.000000cf [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1f0c0 con 0x55f99996a580 -73> 2016-08-24 11:28:37.017959 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138274:17 6.2622d598 201.000000d0 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1f740 con 0x55f99996a580 -72> 2016-08-24 11:28:37.017979 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138274:18 6.aa5d77d8 201.000000d1 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a1fa80 con 0x55f99996a400 -71> 2016-08-24 11:28:37.018009 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138274:19 6.fff6b84b 201.000000d2 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a82000 con 0x55f99996a580 -70> 2016-08-24 11:28:37.134306 7fb2620c0700 1 -- 3.193.148.63:6800/22354 <== osd.8 3.193.149.154:6800/1414 2 ==== osd_op_reply(10 201.000000c9 [read 0~4194304 [fadvise_dontneed]] v0'0 uv4038 ondisk = 0) v7 ==== 132+0+4194304 (2274342180 0 3850264227) 0x55f999a1d6c0 con 0x55f999969380 -69> 2016-08-24 11:28:37.216744 7fb260db4700 1 -- 3.193.148.63:6800/22354 <== osd.9 3.193.149.154:6816/4776 1 ==== osd_op_reply(16 201.000000cf [read 0~4194304 [fadvise_dontneed]] v0'0 uv650 ondisk = 0) v7 ==== 132+0+4194304 (270371816 0 1848106859) 0x55f999a1f0c0 con 0x55f99996a580 -68> 2016-08-24 11:28:37.242659 7fb261bbb700 1 -- 3.193.148.63:6800/22354 <== osd.2 3.193.150.32:6800/23404 2 ==== osd_op_reply(13 201.000000cc [read 0~4194304 [fadvise_dontneed]] v0'0 uv5539 ondisk = 0) v7 ==== 132+0+4194304 (2980121171 0 3567643032) 0x55f999a1ea40 con 0x55f999969b00 -67> 2016-08-24 11:28:37.267594 7fb2621c1700 1 -- 3.193.148.63:6800/22354 <== osd.3 3.193.149.245:6800/7279 4 ==== osd_op_reply(9 201.000000c8 [read 3031877~1162427 [fadvise_dontneed]] v0'0 uv5531 ondisk = 0) v7 ==== 132+0+1162427 (351846972 0 922796555) 0x55f999a1d380 con 0x55f999968f00 -66> 2016-08-24 11:28:37.294168 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138274:20 6.d5d535bb 201.000000d3 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a82340 con 0x55f99996a400 -65> 2016-08-24 11:28:37.318812 7fb260db4700 1 -- 3.193.148.63:6800/22354 <== osd.9 3.193.149.154:6816/4776 2 ==== osd_op_reply(17 201.000000d0 [read 0~4194304 [fadvise_dontneed]] v0'0 uv381 ondisk = 0) v7 ==== 132+0+4194304 (1739907268 0 3938895327) 0x55f999a1f0c0 con 0x55f99996a580 -64> 2016-08-24 11:28:37.319832 7fb261bbb700 1 -- 3.193.148.63:6800/22354 <== osd.2 3.193.150.32:6800/23404 3 ==== osd_op_reply(11 201.000000ca [read 0~4194304 [fadvise_dontneed]] v0'0 uv4103 ondisk = 0) v7 ==== 132+0+4194304 (3983794890 0 1896804775) 0x55f999a1ea40 con 0x55f999969b00 -63> 2016-08-24 11:28:37.392196 7fb260db4700 1 -- 3.193.148.63:6800/22354 <== osd.9 3.193.149.154:6816/4776 3 ==== osd_op_reply(19 201.000000d2 [read 0~4194304 [fadvise_dontneed]] v0'0 uv4019 ondisk = 0) v7 ==== 132+0+4194304 (4290152847 0 4141201163) 0x55f999a1f0c0 con 0x55f99996a580 -62> 2016-08-24 11:28:37.399185 7fb260eb5700 1 -- 3.193.148.63:6800/22354 <== osd.7 3.193.149.154:6804/1537 1 ==== osd_op_reply(12 201.000000cb [read 0~4194304 [fadvise_dontneed]] v0'0 uv1742 ondisk = 0) v7 ==== 132+0+4194304 (761898613 0 3254678077) 0x55f999a1f740 con 0x55f99996a280 -61> 2016-08-24 11:28:37.407705 7fb261aba700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6801/1173 -- osd_op(mds.1.1138274:21 6.b16d511 201.000000d4 [read 0~1652680 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632276) v7 -- ?+0 0x55f999a82680 con 0x55f999969200 -60> 2016-08-24 11:28:37.421438 7fb261cbc700 1 -- 3.193.148.63:6800/22354 <== osd.0 3.193.148.63:6801/1173 2 ==== osd_op_reply(21 201.000000d4 [read 0~1652680 [fadvise_dontneed]] v0'0 uv2195 ondisk = 0) v7 ==== 132+0+1652680 (622877106 0 1949932918) 0x55f999a1e700 con 0x55f999969200 -59> 2016-08-24 11:28:37.550992 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7294,7290 used ino 2000000cd9b but session next is 2000000b24c -58> 2016-08-24 11:28:37.555355 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7296,7290 used ino 2000000cd9c but session next is 2000000b24c -57> 2016-08-24 11:28:37.555412 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7298,7290 used ino 2000000cd9d but session next is 2000000b24c -56> 2016-08-24 11:28:37.555460 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7300,7290 used ino 2000000cd9e but session next is 2000000b24c -55> 2016-08-24 11:28:37.555508 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7302,7290 used ino 2000000cd9f but session next is 2000000b24c -54> 2016-08-24 11:28:37.555555 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7304,7290 used ino 2000000cda0 but session next is 2000000b24c -53> 2016-08-24 11:28:37.555602 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7306,7290 used ino 2000000cda1 but session next is 2000000b24c -52> 2016-08-24 11:28:37.555656 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7308,7290 used ino 2000000cda2 but session next is 2000000b24c -51> 2016-08-24 11:28:37.555714 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7310,7290 used ino 2000000cda3 but session next is 2000000b24c -50> 2016-08-24 11:28:37.555818 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7312,7290 used ino 2000000cda4 but session next is 2000000b24c -49> 2016-08-24 11:28:37.555867 7fb261aba700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7314,7290 used ino 2000000cda5 but session next is 2000000b24c -48> 2016-08-24 11:28:38.673649 7fb266dce700 10 monclient: tick -47> 2016-08-24 11:28:38.673668 7fb266dce700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2016-08-24 11:28:08.673666) -46> 2016-08-24 11:28:38.673683 7fb266dce700 10 log_client log_queue is 11 last_log 11 sent 0 num 11 unsent 11 sending 11 -45> 2016-08-24 11:28:38.673689 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555287 mds.1 3.193.148.63:6800/22354 1 : cluster [WRN] replayed op client.24092763:7294,7290 used ino 2000000cd9b but session next is 2000000b24c -44> 2016-08-24 11:28:38.673708 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555377 mds.1 3.193.148.63:6800/22354 2 : cluster [WRN] replayed op client.24092763:7296,7290 used ino 2000000cd9c but session next is 2000000b24c -43> 2016-08-24 11:28:38.673713 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555432 mds.1 3.193.148.63:6800/22354 3 : cluster [WRN] replayed op client.24092763:7298,7290 used ino 2000000cd9d but session next is 2000000b24c -42> 2016-08-24 11:28:38.673718 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555479 mds.1 3.193.148.63:6800/22354 4 : cluster [WRN] replayed op client.24092763:7300,7290 used ino 2000000cd9e but session next is 2000000b24c -41> 2016-08-24 11:28:38.673723 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555528 mds.1 3.193.148.63:6800/22354 5 : cluster [WRN] replayed op client.24092763:7302,7290 used ino 2000000cd9f but session next is 2000000b24c -40> 2016-08-24 11:28:38.673728 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555575 mds.1 3.193.148.63:6800/22354 6 : cluster [WRN] replayed op client.24092763:7304,7290 used ino 2000000cda0 but session next is 2000000b24c -39> 2016-08-24 11:28:38.673733 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555622 mds.1 3.193.148.63:6800/22354 7 : cluster [WRN] replayed op client.24092763:7306,7290 used ino 2000000cda1 but session next is 2000000b24c -38> 2016-08-24 11:28:38.673737 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555676 mds.1 3.193.148.63:6800/22354 8 : cluster [WRN] replayed op client.24092763:7308,7290 used ino 2000000cda2 but session next is 2000000b24c -37> 2016-08-24 11:28:38.673742 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555733 mds.1 3.193.148.63:6800/22354 9 : cluster [WRN] replayed op client.24092763:7310,7290 used ino 2000000cda3 but session next is 2000000b24c -36> 2016-08-24 11:28:38.673746 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555839 mds.1 3.193.148.63:6800/22354 10 : cluster [WRN] replayed op client.24092763:7312,7290 used ino 2000000cda4 but session next is 2000000b24c -35> 2016-08-24 11:28:38.673751 7fb266dce700 10 log_client will send 2016-08-24 11:28:37.555889 mds.1 3.193.148.63:6800/22354 11 : cluster [WRN] replayed op client.24092763:7314,7290 used ino 2000000cda5 but session next is 2000000b24c -34> 2016-08-24 11:28:38.673758 7fb266dce700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -33> 2016-08-24 11:28:38.673762 7fb266dce700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- log(11 entries from seq 1 at 2016-08-24 11:28:37.555287) v1 -- ?+0 0x55f99994e080 con 0x55f999968300 -32> 2016-08-24 11:28:38.673956 7fb265dcc700 2 -- 3.193.148.63:6800/22354 >> 3.193.148.63:6789/0 pipe(0x55f9999fe000 sd=8 :33246 s=2 pgs=6959240 cs=1 l=1 c=0x55f999968300).reader got KEEPALIVE_ACK -31> 2016-08-24 11:28:39.605939 7fb260fb6700 1 -- 3.193.148.63:6800/22354 <== osd.1 3.193.151.182:6800/1753 1 ==== osd_op_reply(18 201.000000d1 [read 0~4194304 [fadvise_dontneed]] v0'0 uv5492 ondisk = 0) v7 ==== 132+0+4194304 (2309471830 0 4190685855) 0x55f999a1cd00 con 0x55f99996a400 -30> 2016-08-24 11:28:39.692493 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 13 ==== log(last 11) v1 ==== 28+0+0 (2678348754 0 0) 0x55f999a8afc0 con 0x55f999968300 -29> 2016-08-24 11:28:39.692514 7fb2685d1700 10 log_client handle_log_ack log(last 11) v1 -28> 2016-08-24 11:28:39.692516 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555287 mds.1 3.193.148.63:6800/22354 1 : cluster [WRN] replayed op client.24092763:7294,7290 used ino 2000000cd9b but session next is 2000000b24c -27> 2016-08-24 11:28:39.692527 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555377 mds.1 3.193.148.63:6800/22354 2 : cluster [WRN] replayed op client.24092763:7296,7290 used ino 2000000cd9c but session next is 2000000b24c -26> 2016-08-24 11:28:39.692536 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555432 mds.1 3.193.148.63:6800/22354 3 : cluster [WRN] replayed op client.24092763:7298,7290 used ino 2000000cd9d but session next is 2000000b24c -25> 2016-08-24 11:28:39.692542 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555479 mds.1 3.193.148.63:6800/22354 4 : cluster [WRN] replayed op client.24092763:7300,7290 used ino 2000000cd9e but session next is 2000000b24c -24> 2016-08-24 11:28:39.692553 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555528 mds.1 3.193.148.63:6800/22354 5 : cluster [WRN] replayed op client.24092763:7302,7290 used ino 2000000cd9f but session next is 2000000b24c -23> 2016-08-24 11:28:39.692560 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555575 mds.1 3.193.148.63:6800/22354 6 : cluster [WRN] replayed op client.24092763:7304,7290 used ino 2000000cda0 but session next is 2000000b24c -22> 2016-08-24 11:28:39.692566 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555622 mds.1 3.193.148.63:6800/22354 7 : cluster [WRN] replayed op client.24092763:7306,7290 used ino 2000000cda1 but session next is 2000000b24c -21> 2016-08-24 11:28:39.692572 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555676 mds.1 3.193.148.63:6800/22354 8 : cluster [WRN] replayed op client.24092763:7308,7290 used ino 2000000cda2 but session next is 2000000b24c -20> 2016-08-24 11:28:39.692580 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555733 mds.1 3.193.148.63:6800/22354 9 : cluster [WRN] replayed op client.24092763:7310,7290 used ino 2000000cda3 but session next is 2000000b24c -19> 2016-08-24 11:28:39.692587 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555839 mds.1 3.193.148.63:6800/22354 10 : cluster [WRN] replayed op client.24092763:7312,7290 used ino 2000000cda4 but session next is 2000000b24c -18> 2016-08-24 11:28:39.692593 7fb2685d1700 10 log_client logged 2016-08-24 11:28:37.555889 mds.1 3.193.148.63:6800/22354 11 : cluster [WRN] replayed op client.24092763:7314,7290 used ino 2000000cda5 but session next is 2000000b24c -17> 2016-08-24 11:28:39.746593 7fb2654ca700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -16> 2016-08-24 11:28:39.746607 7fb2654ca700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mdsbeacon(24534054/switcheroo up:replay seq 2 v1138274) v7 -- ?+0 0x55f999a29180 con 0x55f999968300 -15> 2016-08-24 11:28:39.746862 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 14 ==== mdsbeacon(24534054/switcheroo up:replay seq 2 v1138274) v7 ==== 135+0+0 (3483662910 0 0) 0x55f999a28000 con 0x55f999968300 -14> 2016-08-24 11:28:40.740970 7fb260fb6700 1 -- 3.193.148.63:6800/22354 <== osd.1 3.193.151.182:6800/1753 2 ==== osd_op_reply(15 201.000000ce [read 0~4194304 [fadvise_dontneed]] v0'0 uv596 ondisk = 0) v7 ==== 132+0+4194304 (270128439 0 45223128) 0x55f999a1cd00 con 0x55f99996a400 -13> 2016-08-24 11:28:43.746651 7fb2654ca700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -12> 2016-08-24 11:28:43.746672 7fb2654ca700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mdsbeacon(24534054/switcheroo up:replay seq 3 v1138274) v7 -- ?+0 0x55f999a29500 con 0x55f999968300 -11> 2016-08-24 11:28:43.746972 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 15 ==== mdsbeacon(24534054/switcheroo up:replay seq 3 v1138274) v7 ==== 135+0+0 (2379630295 0 0) 0x55f999a29180 con 0x55f999968300 -10> 2016-08-24 11:28:44.645296 7fb260fb6700 1 -- 3.193.148.63:6800/22354 <== osd.1 3.193.151.182:6800/1753 3 ==== osd_op_reply(14 201.000000cd [read 0~4194304 [fadvise_dontneed]] v0'0 uv6284 ondisk = 0) v7 ==== 132+0+4194304 (4128806170 0 1824143700) 0x55f999a1cd00 con 0x55f99996a400 -9> 2016-08-24 11:28:45.027028 7fb261aba700 -1 mds/journal.cc: In function 'virtual void EImportStart::replay(MDSRank*)' thread 7fb261aba700 time 2016-08-24 11:28:45.025881 mds/journal.cc: 2929: FAILED assert(mds->sessionmap.get_version() == cmapv) ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55f98ea972a0] 2: (EImportStart::replay(MDSRank*)+0x9e5) [0x55f98e970e75] 3: (MDLog::_replay_thread()+0xe73) [0x55f98e8efb23] 4: (MDLog::ReplayThread::entry()+0xd) [0x55f98e69a84d] 5: (()+0x75ca) [0x7fb26dab85ca] 6: (clone()+0x6d) [0x7fb26c4f8f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. -8> 2016-08-24 11:28:45.731924 7fb260fb6700 1 -- 3.193.148.63:6800/22354 <== osd.1 3.193.151.182:6800/1753 4 ==== osd_op_reply(20 201.000000d3 [read 0~4194304 [fadvise_dontneed]] v0'0 uv6451 ondisk = 0) v7 ==== 132+0+4194304 (2291321620 0 2327309785) 0x55f999a1cd00 con 0x55f99996a400 -7> 2016-08-24 11:28:46.022025 7fb2685d1700 1 -- 3.193.148.63:6800/22354 <== mon.0 3.193.148.63:6789/0 16 ==== osd_map(632277..632277 src has 631672..632277) v3 ==== 350+0+0 (3901410990 0 0) 0x55f99994d900 con 0x55f999968300 -6> 2016-08-24 11:28:47.746733 7fb2654ca700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -5> 2016-08-24 11:28:47.746749 7fb2654ca700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mdsbeacon(24534054/switcheroo up:replay seq 4 v1138274) v7 -- ?+0 0x55f999a29880 con 0x55f999968300 -4> 2016-08-24 11:28:48.673839 7fb266dce700 10 monclient: tick -3> 2016-08-24 11:28:48.673861 7fb266dce700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2016-08-24 11:28:18.673857) -2> 2016-08-24 11:28:48.674037 7fb265dcc700 2 -- 3.193.148.63:6800/22354 >> 3.193.148.63:6789/0 pipe(0x55f9999fe000 sd=8 :33246 s=2 pgs=6959240 cs=1 l=1 c=0x55f999968300).reader got KEEPALIVE_ACK -1> 2016-08-24 11:28:51.746782 7fb2654ca700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 0> 2016-08-24 11:28:51.746797 7fb2654ca700 1 -- 3.193.148.63:6800/22354 --> 3.193.148.63:6789/0 -- mdsbeacon(24534054/switcheroo up:replay seq 5 v1138274) v7 -- ?+0 0x55f999a2a680 con 0x55f999968300 --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 kinetic 1/ 5 fuse -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.switcheroo.log --- end dump of recent events --- 2016-08-24 11:28:53.976192 7fb261aba700 -1 *** Caught signal (Aborted) ** in thread 7fb261aba700 thread_name:md_log_replay ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (()+0x514a2e) [0x55f98e989a2e] 2: (()+0x10c30) [0x7fb26dac1c30] 3: (gsignal()+0x35) [0x7fb26c42a6f5] 4: (abort()+0x16a) [0x7fb26c42c2fa] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55f98ea9748b] 6: (EImportStart::replay(MDSRank*)+0x9e5) [0x55f98e970e75] 7: (MDLog::_replay_thread()+0xe73) [0x55f98e8efb23] 8: (MDLog::ReplayThread::entry()+0xd) [0x55f98e69a84d] 9: (()+0x75ca) [0x7fb26dab85ca] 10: (clone()+0x6d) [0x7fb26c4f8f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2016-08-24 11:28:53.976192 7fb261aba700 -1 *** Caught signal (Aborted) ** in thread 7fb261aba700 thread_name:md_log_replay ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (()+0x514a2e) [0x55f98e989a2e] 2: (()+0x10c30) [0x7fb26dac1c30] 3: (gsignal()+0x35) [0x7fb26c42a6f5] 4: (abort()+0x16a) [0x7fb26c42c2fa] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55f98ea9748b] 6: (EImportStart::replay(MDSRank*)+0x9e5) [0x55f98e970e75] 7: (MDLog::_replay_thread()+0xe73) [0x55f98e8efb23] 8: (MDLog::ReplayThread::entry()+0xd) [0x55f98e69a84d] 9: (()+0x75ca) [0x7fb26dab85ca] 10: (clone()+0x6d) [0x7fb26c4f8f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 kinetic 1/ 5 fuse -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.switcheroo.log --- end dump of recent events --- 2016-08-24 11:28:54.163403 7f6cf6292200 0 set uid:gid to 167:167 (ceph:ceph) 2016-08-24 11:28:54.163450 7f6cf6292200 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mds, pid 22434 2016-08-24 11:28:54.163709 7f6cf6292200 0 pidfile_write: ignore empty --pid-file 2016-08-24 11:28:54.545249 7f6cf01ba700 1 mds.switcheroo handle_mds_map standby 2016-08-24 11:28:54.619237 7f6cf01ba700 1 mds.1.1138276 handle_mds_map i am now mds.1.1138276 2016-08-24 11:28:54.619241 7f6cf01ba700 1 mds.1.1138276 handle_mds_map state change up:boot --> up:replay 2016-08-24 11:28:54.619250 7f6cf01ba700 1 mds.1.1138276 replay_start 2016-08-24 11:28:54.619254 7f6cf01ba700 1 mds.1.1138276 recovery set is 0 2016-08-24 11:28:54.619262 7f6cf01ba700 1 mds.1.1138276 waiting for osdmap 632278 (which blacklists prior instance) 2016-08-24 11:28:54.771947 7f6ceb8b0700 0 mds.1.cache creating system inode with ino:101 2016-08-24 11:28:54.772074 7f6ceb8b0700 0 mds.1.cache creating system inode with ino:1 2016-08-24 11:28:55.217928 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7294,7290 used ino 2000000cd9b but session next is 2000000b24c 2016-08-24 11:28:55.218006 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7296,7290 used ino 2000000cd9c but session next is 2000000b24c 2016-08-24 11:28:55.218042 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7298,7290 used ino 2000000cd9d but session next is 2000000b24c 2016-08-24 11:28:55.218071 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7300,7290 used ino 2000000cd9e but session next is 2000000b24c 2016-08-24 11:28:55.218109 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7302,7290 used ino 2000000cd9f but session next is 2000000b24c 2016-08-24 11:28:55.218139 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7304,7290 used ino 2000000cda0 but session next is 2000000b24c 2016-08-24 11:28:55.218168 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7306,7290 used ino 2000000cda1 but session next is 2000000b24c 2016-08-24 11:28:55.218197 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7308,7290 used ino 2000000cda2 but session next is 2000000b24c 2016-08-24 11:28:55.218251 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7310,7290 used ino 2000000cda3 but session next is 2000000b24c 2016-08-24 11:28:55.218338 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7312,7290 used ino 2000000cda4 but session next is 2000000b24c 2016-08-24 11:28:55.218369 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7314,7290 used ino 2000000cda5 but session next is 2000000b24c 2016-08-24 11:28:58.159406 7f6ce96a3700 -1 mds/journal.cc: In function 'virtual void EImportStart::replay(MDSRank*)' thread 7f6ce96a3700 time 2016-08-24 11:28:58.158246 mds/journal.cc: 2929: FAILED assert(mds->sessionmap.get_version() == cmapv) ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x555dddff22a0] 2: (EImportStart::replay(MDSRank*)+0x9e5) [0x555dddecbe75] 3: (MDLog::_replay_thread()+0xe73) [0x555ddde4ab23] 4: (MDLog::ReplayThread::entry()+0xd) [0x555dddbf584d] 5: (()+0x75ca) [0x7f6cf56a15ca] 6: (clone()+0x6d) [0x7f6cf40e1f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -206> 2016-08-24 11:28:54.157323 7f6cf6292200 5 asok(0x555de8ef2000) register_command perfcounters_dump hook 0x555de8ed6030 -205> 2016-08-24 11:28:54.157343 7f6cf6292200 5 asok(0x555de8ef2000) register_command 1 hook 0x555de8ed6030 -204> 2016-08-24 11:28:54.157348 7f6cf6292200 5 asok(0x555de8ef2000) register_command perf dump hook 0x555de8ed6030 -203> 2016-08-24 11:28:54.157353 7f6cf6292200 5 asok(0x555de8ef2000) register_command perfcounters_schema hook 0x555de8ed6030 -202> 2016-08-24 11:28:54.157358 7f6cf6292200 5 asok(0x555de8ef2000) register_command 2 hook 0x555de8ed6030 -201> 2016-08-24 11:28:54.157361 7f6cf6292200 5 asok(0x555de8ef2000) register_command perf schema hook 0x555de8ed6030 -200> 2016-08-24 11:28:54.157364 7f6cf6292200 5 asok(0x555de8ef2000) register_command perf reset hook 0x555de8ed6030 -199> 2016-08-24 11:28:54.157369 7f6cf6292200 5 asok(0x555de8ef2000) register_command config show hook 0x555de8ed6030 -198> 2016-08-24 11:28:54.157372 7f6cf6292200 5 asok(0x555de8ef2000) register_command config set hook 0x555de8ed6030 -197> 2016-08-24 11:28:54.157376 7f6cf6292200 5 asok(0x555de8ef2000) register_command config get hook 0x555de8ed6030 -196> 2016-08-24 11:28:54.157381 7f6cf6292200 5 asok(0x555de8ef2000) register_command config diff hook 0x555de8ed6030 -195> 2016-08-24 11:28:54.157385 7f6cf6292200 5 asok(0x555de8ef2000) register_command log flush hook 0x555de8ed6030 -194> 2016-08-24 11:28:54.157390 7f6cf6292200 5 asok(0x555de8ef2000) register_command log dump hook 0x555de8ed6030 -193> 2016-08-24 11:28:54.157395 7f6cf6292200 5 asok(0x555de8ef2000) register_command log reopen hook 0x555de8ed6030 -192> 2016-08-24 11:28:54.163403 7f6cf6292200 0 set uid:gid to 167:167 (ceph:ceph) -191> 2016-08-24 11:28:54.163450 7f6cf6292200 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mds, pid 22434 -190> 2016-08-24 11:28:54.163691 7f6cf6292200 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/22434 need_addr=1 -189> 2016-08-24 11:28:54.163709 7f6cf6292200 0 pidfile_write: ignore empty --pid-file -188> 2016-08-24 11:28:54.165972 7f6cf6292200 5 asok(0x555de8ef2000) init /var/run/ceph/ceph-mds.switcheroo.asok -187> 2016-08-24 11:28:54.165998 7f6cf6292200 5 asok(0x555de8ef2000) bind_and_listen /var/run/ceph/ceph-mds.switcheroo.asok -186> 2016-08-24 11:28:54.166198 7f6cf6292200 5 asok(0x555de8ef2000) register_command 0 hook 0x555de8ed40a8 -185> 2016-08-24 11:28:54.166213 7f6cf6292200 5 asok(0x555de8ef2000) register_command version hook 0x555de8ed40a8 -184> 2016-08-24 11:28:54.166224 7f6cf6292200 5 asok(0x555de8ef2000) register_command git_version hook 0x555de8ed40a8 -183> 2016-08-24 11:28:54.166252 7f6cf6292200 5 asok(0x555de8ef2000) register_command help hook 0x555de8ed6130 -182> 2016-08-24 11:28:54.166264 7f6cf6292200 5 asok(0x555de8ef2000) register_command get_command_descriptions hook 0x555de8ed6140 -181> 2016-08-24 11:28:54.166307 7f6cf6292200 10 monclient(hunting): build_initial_monmap -180> 2016-08-24 11:28:54.166308 7f6cf19bd700 5 asok(0x555de8ef2000) entry start -179> 2016-08-24 11:28:54.166342 7f6cf6292200 1 -- 0.0.0.0:6800/22434 messenger.start -178> 2016-08-24 11:28:54.166386 7f6cf6292200 5 adding auth protocol: cephx -177> 2016-08-24 11:28:54.166398 7f6cf6292200 5 adding auth protocol: cephx -176> 2016-08-24 11:28:54.166463 7f6cf6292200 5 asok(0x555de8ef2000) register_command objecter_requests hook 0x555de8ed6150 -175> 2016-08-24 11:28:54.166532 7f6cf6292200 1 accepter.accepter.start -174> 2016-08-24 11:28:54.166562 7f6cf6292200 10 monclient(hunting): init -173> 2016-08-24 11:28:54.166574 7f6cf6292200 5 adding auth protocol: cephx -172> 2016-08-24 11:28:54.166583 7f6cf6292200 10 monclient(hunting): auth_supported 2 method cephx -171> 2016-08-24 11:28:54.166758 7f6cf6292200 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-switcheroo/keyring -170> 2016-08-24 11:28:54.166820 7f6cf6292200 10 monclient(hunting): _reopen_session rank -1 name -169> 2016-08-24 11:28:54.166972 7f6cf6292200 10 monclient(hunting): picked mon.noname-a con 0x555de8f26300 addr 3.193.148.63:6789/0 -168> 2016-08-24 11:28:54.166992 7f6cf6292200 10 monclient(hunting): _send_mon_message to mon.noname-a at 3.193.148.63:6789/0 -167> 2016-08-24 11:28:54.169245 7f6cf6292200 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- auth(proto 0 35 bytes epoch 0) v1 -- ?+0 0x555de8f0aa00 con 0x555de8f26300 -166> 2016-08-24 11:28:54.169255 7f6cf6292200 10 monclient(hunting): renew_subs -165> 2016-08-24 11:28:54.168619 7f6cf628c700 1 -- 3.193.148.63:6800/22434 learned my addr 3.193.148.63:6800/22434 -164> 2016-08-24 11:28:54.170416 7f6ced9b5700 2 -- 3.193.148.63:6800/22434 >> 3.193.148.63:6789/0 pipe(0x555de8fbc000 sd=8 :33264 s=2 pgs=6959244 cs=1 l=1 c=0x555de8f26300).reader got KEEPALIVE_ACK -163> 2016-08-24 11:28:54.170546 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 1 ==== mon_map magic: 0 v1 ==== 503+0+0 (716945685 0 0) 0x555de8eee900 con 0x555de8f26300 -162> 2016-08-24 11:28:54.170560 7f6cf01ba700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -161> 2016-08-24 11:28:54.170569 7f6cf01ba700 10 monclient(hunting): got monmap 5, mon.noname-a is now rank -1 -160> 2016-08-24 11:28:54.170571 7f6cf01ba700 10 monclient(hunting): dump: epoch 5 fsid 40216c8a-ba3c-4cec-9e80-212396e214a1 last_changed 2016-05-05 10:18:29.089450 created 0.000000 0: 3.193.148.63:6789/0 mon.switcheroo 1: 3.193.149.245:6789/0 mon.microstation 2: 3.193.150.32:6789/0 mon.dashboardpc -159> 2016-08-24 11:28:54.170592 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3647111377 0 0) 0x555de8f0b180 con 0x555de8f26300 -158> 2016-08-24 11:28:54.170607 7f6cf01ba700 10 monclient(hunting): my global_id is 24534066 -157> 2016-08-24 11:28:54.170685 7f6cf01ba700 10 monclient(hunting): _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -156> 2016-08-24 11:28:54.170690 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x555de8f0af00 con 0x555de8f26300 -155> 2016-08-24 11:28:54.175244 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (3309886966 0 0) 0x555de8f0b400 con 0x555de8f26300 -154> 2016-08-24 11:28:54.175314 7f6cf01ba700 10 monclient(hunting): _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -153> 2016-08-24 11:28:54.175319 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0 0x555de8f0b180 con 0x555de8f26300 -152> 2016-08-24 11:28:54.175932 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (2303073522 0 0) 0x555de8f0aa00 con 0x555de8f26300 -151> 2016-08-24 11:28:54.176041 7f6cf01ba700 1 monclient(hunting): found mon.switcheroo -150> 2016-08-24 11:28:54.176043 7f6cf01ba700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -149> 2016-08-24 11:28:54.176047 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x555de8ee3000 con 0x555de8f26300 -148> 2016-08-24 11:28:54.176061 7f6cf01ba700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2016-08-24 11:28:24.176060) -147> 2016-08-24 11:28:54.176068 7f6cf01ba700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -146> 2016-08-24 11:28:54.176071 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x555de8f0b400 con 0x555de8f26300 -145> 2016-08-24 11:28:54.176089 7f6cf6292200 5 monclient: authenticate success, global_id 24534066 -144> 2016-08-24 11:28:54.176101 7f6cf6292200 10 monclient: wait_auth_rotating waiting (until 2016-08-24 11:29:24.176101) -143> 2016-08-24 11:28:54.177508 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 503+0+0 (716945685 0 0) 0x555de8eeeb40 con 0x555de8f26300 -142> 2016-08-24 11:28:54.177520 7f6cf01ba700 10 monclient: handle_monmap mon_map magic: 0 v1 -141> 2016-08-24 11:28:54.177528 7f6cf01ba700 10 monclient: got monmap 5, mon.switcheroo is now rank 0 -140> 2016-08-24 11:28:54.177530 7f6cf01ba700 10 monclient: dump: epoch 5 fsid 40216c8a-ba3c-4cec-9e80-212396e214a1 last_changed 2016-05-05 10:18:29.089450 created 0.000000 0: 3.193.148.63:6789/0 mon.switcheroo 1: 3.193.149.245:6789/0 mon.microstation 2: 3.193.150.32:6789/0 mon.dashboardpc -139> 2016-08-24 11:28:54.177544 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (3541963397 0 0) 0x555de8f0b680 con 0x555de8f26300 -138> 2016-08-24 11:28:54.177619 7f6cf01ba700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2016-08-24 11:28:24.177619) -137> 2016-08-24 11:28:54.177635 7f6cf6292200 10 monclient: wait_auth_rotating done -136> 2016-08-24 11:28:54.177678 7f6cf6292200 10 monclient: renew_subs -135> 2016-08-24 11:28:54.177689 7f6cf6292200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -134> 2016-08-24 11:28:54.177700 7f6cf6292200 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x555de8ee3200 con 0x555de8f26300 -133> 2016-08-24 11:28:54.177717 7f6cf6292200 10 monclient: renew_subs -132> 2016-08-24 11:28:54.177727 7f6cf6292200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -131> 2016-08-24 11:28:54.177738 7f6cf6292200 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- mon_subscribe({mdsmap=0+}) v2 -- ?+0 0x555de8ee3400 con 0x555de8f26300 -130> 2016-08-24 11:28:54.178251 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 7 ==== osd_map(632277..632277 src has 631672..632277) v3 ==== 32550+0+0 (4029138412 0 0) 0x555de8f0aa00 con 0x555de8f26300 -129> 2016-08-24 11:28:54.178459 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 8 ==== mdsmap(e 1138274) v1 ==== 462+0+0 (472034357 0 0) 0x555de8eeefc0 con 0x555de8f26300 -128> 2016-08-24 11:28:54.178468 7f6cf01ba700 5 mds.switcheroo handle_mds_map epoch 1138274 from mon.0 -127> 2016-08-24 11:28:54.178506 7f6cf6292200 5 asok(0x555de8ef2000) register_command status hook 0x555de8ed6190 -126> 2016-08-24 11:28:54.178521 7f6cf6292200 5 asok(0x555de8ef2000) register_command dump_ops_in_flight hook 0x555de8ed6190 -125> 2016-08-24 11:28:54.178533 7f6cf6292200 5 asok(0x555de8ef2000) register_command ops hook 0x555de8ed6190 -124> 2016-08-24 11:28:54.178544 7f6cf6292200 5 asok(0x555de8ef2000) register_command dump_blocked_ops hook 0x555de8ed6190 -123> 2016-08-24 11:28:54.178556 7f6cf6292200 5 asok(0x555de8ef2000) register_command dump_historic_ops hook 0x555de8ed6190 -122> 2016-08-24 11:28:54.178568 7f6cf6292200 5 asok(0x555de8ef2000) register_command scrub_path hook 0x555de8ed6190 -121> 2016-08-24 11:28:54.178580 7f6cf6292200 5 asok(0x555de8ef2000) register_command tag path hook 0x555de8ed6190 -120> 2016-08-24 11:28:54.178590 7f6cf6292200 5 asok(0x555de8ef2000) register_command flush_path hook 0x555de8ed6190 -119> 2016-08-24 11:28:54.178601 7f6cf6292200 5 asok(0x555de8ef2000) register_command export dir hook 0x555de8ed6190 -118> 2016-08-24 11:28:54.178612 7f6cf6292200 5 asok(0x555de8ef2000) register_command dump cache hook 0x555de8ed6190 -117> 2016-08-24 11:28:54.178624 7f6cf6292200 5 asok(0x555de8ef2000) register_command session evict hook 0x555de8ed6190 -116> 2016-08-24 11:28:54.178635 7f6cf6292200 5 asok(0x555de8ef2000) register_command osdmap barrier hook 0x555de8ed6190 -115> 2016-08-24 11:28:54.178647 7f6cf6292200 5 asok(0x555de8ef2000) register_command session ls hook 0x555de8ed6190 -114> 2016-08-24 11:28:54.178659 7f6cf6292200 5 asok(0x555de8ef2000) register_command flush journal hook 0x555de8ed6190 -113> 2016-08-24 11:28:54.178670 7f6cf6292200 5 asok(0x555de8ef2000) register_command force_readonly hook 0x555de8ed6190 -112> 2016-08-24 11:28:54.178681 7f6cf6292200 5 asok(0x555de8ef2000) register_command get subtrees hook 0x555de8ed6190 -111> 2016-08-24 11:28:54.178692 7f6cf6292200 5 asok(0x555de8ef2000) register_command dirfrag split hook 0x555de8ed6190 -110> 2016-08-24 11:28:54.178704 7f6cf6292200 5 asok(0x555de8ef2000) register_command dirfrag merge hook 0x555de8ed6190 -109> 2016-08-24 11:28:54.178715 7f6cf6292200 5 asok(0x555de8ef2000) register_command dirfrag ls hook 0x555de8ed6190 -108> 2016-08-24 11:28:54.249732 7f6cf6292200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -107> 2016-08-24 11:28:54.249762 7f6cf6292200 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- mdsbeacon(24534066/switcheroo up:boot seq 1 v1138274) v7 -- ?+0 0x555de8fe6000 con 0x555de8f26300 -106> 2016-08-24 11:28:54.545171 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 9 ==== mdsmap(e 1138275) v1 ==== 687+0+0 (2565360266 0 0) 0x555de8eef200 con 0x555de8f26300 -105> 2016-08-24 11:28:54.545198 7f6cf01ba700 5 mds.switcheroo handle_mds_map epoch 1138275 from mon.0 -104> 2016-08-24 11:28:54.545249 7f6cf01ba700 1 mds.switcheroo handle_mds_map standby -103> 2016-08-24 11:28:54.545259 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 10 ==== mdsbeacon(24534066/switcheroo up:boot seq 1 v1138275) v7 ==== 139+0+0 (2999228297 0 0) 0x555de8fe6380 con 0x555de8f26300 -102> 2016-08-24 11:28:54.618946 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 11 ==== mdsmap(e 1138276) v1 ==== 968+0+0 (1751275336 0 0) 0x555de8eef440 con 0x555de8f26300 -101> 2016-08-24 11:28:54.618970 7f6cf01ba700 5 mds.switcheroo handle_mds_map epoch 1138276 from mon.0 -100> 2016-08-24 11:28:54.619081 7f6cf01ba700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -99> 2016-08-24 11:28:54.619124 7f6cf01ba700 10 monclient: renew_subs -98> 2016-08-24 11:28:54.619130 7f6cf01ba700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -97> 2016-08-24 11:28:54.619140 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=632278}) v2 -- ?+0 0x555de8ee3000 con 0x555de8f26300 -96> 2016-08-24 11:28:54.619237 7f6cf01ba700 1 mds.1.1138276 handle_mds_map i am now mds.1.1138276 -95> 2016-08-24 11:28:54.619241 7f6cf01ba700 1 mds.1.1138276 handle_mds_map state change up:boot --> up:replay -94> 2016-08-24 11:28:54.619250 7f6cf01ba700 1 mds.1.1138276 replay_start -93> 2016-08-24 11:28:54.619254 7f6cf01ba700 1 mds.1.1138276 recovery set is 0 -92> 2016-08-24 11:28:54.619262 7f6cf01ba700 1 mds.1.1138276 waiting for osdmap 632278 (which blacklists prior instance) -91> 2016-08-24 11:28:54.619574 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 <== mon.0 3.193.148.63:6789/0 12 ==== osd_map(632278..632278 src has 631672..632278) v3 ==== 358+0+0 (3195025293 0 0) 0x555de8f0af00 con 0x555de8f26300 -90> 2016-08-24 11:28:54.619625 7f6cf01ba700 10 monclient: renew_subs -89> 2016-08-24 11:28:54.619628 7f6cf01ba700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -88> 2016-08-24 11:28:54.619631 7f6cf01ba700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=632279}) v2 -- ?+0 0x555de8ee3200 con 0x555de8f26300 -87> 2016-08-24 11:28:54.619684 7f6ceb8b0700 2 mds.1.1138276 boot_start 0: opening inotable -86> 2016-08-24 11:28:54.619799 7f6ceb8b0700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138276:1 6.ad6e75db mds1_inotable [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fda680 con 0x555de8f26a80 -85> 2016-08-24 11:28:54.619830 7f6ceb8b0700 2 mds.1.1138276 boot_start 0: opening sessionmap -84> 2016-08-24 11:28:54.619890 7f6ceb8b0700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.154:6812/2220 -- osd_op(mds.1.1138276:2 6.32a2fe43 mds1_sessionmap [omap-get-header 0~0,omap-get-vals 0~16] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fda9c0 con 0x555de8f26c00 -83> 2016-08-24 11:28:54.619918 7f6ceb8b0700 2 mds.1.1138276 boot_start 0: opening mds log -82> 2016-08-24 11:28:54.619929 7f6ceb8b0700 5 mds.1.log open discovering log bounds -81> 2016-08-24 11:28:54.620002 7f6ceaead700 4 mds.1.journalpointer Reading journal pointer '401.00000000' -80> 2016-08-24 11:28:54.620070 7f6ceaead700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.154:6800/1414 -- osd_op(mds.1.1138276:3 6.fcd6c765 401.00000000 [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdb040 con 0x555de8f26f00 -79> 2016-08-24 11:28:54.623377 7f6ce9ba8700 1 -- 3.193.148.63:6800/22434 <== osd.8 3.193.149.154:6800/1414 1 ==== osd_op_reply(3 401.00000000 [read 0~22] v0'0 uv1 ondisk = 0) v7 ==== 132+0+22 (4214892891 0 369460326) 0x555de8fdb380 con 0x555de8f26f00 -78> 2016-08-24 11:28:54.623429 7f6ceaead700 1 mds.1.journaler(ro) recover start -77> 2016-08-24 11:28:54.623440 7f6ceaead700 1 mds.1.journaler(ro) read_head -76> 2016-08-24 11:28:54.623470 7f6ceaead700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.154:6812/2220 -- osd_op(mds.1.1138276:4 6.5664a157 201.00000000 [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdb6c0 con 0x555de8f26c00 -75> 2016-08-24 11:28:54.623493 7f6ceaead700 4 mds.1.log Waiting for journal 201 to recover... -74> 2016-08-24 11:28:54.650803 7f6ce9ca9700 1 -- 3.193.148.63:6800/22434 <== osd.6 3.193.149.154:6812/2220 1 ==== osd_op_reply(4 201.00000000 [read 0~90] v0'0 uv20080 ondisk = 0) v7 ==== 132+0+90 (1067775253 0 3155850792) 0x555de8fda9c0 con 0x555de8f26c00 -73> 2016-08-24 11:28:54.650843 7f6ceb8b0700 1 mds.1.journaler(ro) _finish_read_head loghead(trim 838860800, expire 841892677, write 890374656, stream_format 1). probing for end of log (from 890374656)... -72> 2016-08-24 11:28:54.650849 7f6ceb8b0700 1 mds.1.journaler(ro) probing for end of the log -71> 2016-08-24 11:28:54.650938 7f6ceb8b0700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6801/1173 -- osd_op(mds.1.1138276:5 6.b16d511 201.000000d4 [stat] snapc 0=[] ack+read+rwordered+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdad00 con 0x555de8f26d80 -70> 2016-08-24 11:28:54.651008 7f6ceb8b0700 1 -- 3.193.148.63:6800/22434 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138276:6 6.3071b1a0 201.000000d5 [stat] snapc 0=[] ack+read+rwordered+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdb6c0 con 0x555de8f27800 -69> 2016-08-24 11:28:54.652051 7f6ce9ca9700 1 -- 3.193.148.63:6800/22434 <== osd.6 3.193.149.154:6812/2220 2 ==== osd_op_reply(2 mds1_sessionmap [omap-get-header 0~0,omap-get-vals 0~16] v0'0 uv2124 ondisk = 0) v7 ==== 177+0+807 (447421225 0 2854467212) 0x555de8fda9c0 con 0x555de8f26c00 -68> 2016-08-24 11:28:54.653324 7f6ce97a4700 1 -- 3.193.148.63:6800/22434 <== osd.2 3.193.150.32:6800/23404 1 ==== osd_op_reply(6 201.000000d5 [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 (4091682702 0 0) 0x555de8fdc3c0 con 0x555de8f27800 -67> 2016-08-24 11:28:54.677357 7f6ce98a5700 1 -- 3.193.148.63:6800/22434 <== osd.0 3.193.148.63:6801/1173 1 ==== osd_op_reply(5 201.000000d4 [stat] v0'0 uv2195 ondisk = 0) v7 ==== 132+0+16 (1810834021 0 2553466074) 0x555de8fdc700 con 0x555de8f26d80 -66> 2016-08-24 11:28:54.677430 7f6ceb8b0700 1 mds.1.journaler(ro) _finish_probe_end write_pos = 890845128 (header had 890374656). recovered. -65> 2016-08-24 11:28:54.677447 7f6ceaead700 4 mds.1.log Journal 201 recovered. -64> 2016-08-24 11:28:54.677450 7f6ceaead700 4 mds.1.log Recovered journal 201 in format 1 -63> 2016-08-24 11:28:54.771869 7f6ce9daa700 1 -- 3.193.148.63:6800/22434 <== osd.3 3.193.149.245:6800/7279 1 ==== osd_op_reply(1 mds1_inotable [read 0~98] v0'0 uv3885 ondisk = 0) v7 ==== 133+0+98 (470069693 0 3676677995) 0x555de8fdba00 con 0x555de8f26a80 -62> 2016-08-24 11:28:54.771934 7f6ceb8b0700 2 mds.1.1138276 boot_start 1: loading/discovering base inodes -61> 2016-08-24 11:28:54.771947 7f6ceb8b0700 0 mds.1.cache creating system inode with ino:101 -60> 2016-08-24 11:28:54.772029 7f6ceb8b0700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138276:7 6.75b54883 101.00000000 [getxattr inode] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdbd40 con 0x555de8f26a80 -59> 2016-08-24 11:28:54.772064 7f6ceb8b0700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138276:8 6.2fd1f055 101.00000000.inode [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdc080 con 0x555de8f26a80 -58> 2016-08-24 11:28:54.772074 7f6ceb8b0700 0 mds.1.cache creating system inode with ino:1 -57> 2016-08-24 11:28:54.773350 7f6ce9daa700 1 -- 3.193.148.63:6800/22434 <== osd.3 3.193.149.245:6800/7279 2 ==== osd_op_reply(8 101.00000000.inode [read 0~508] v0'0 uv42 ondisk = 0) v7 ==== 138+0+508 (108667050 0 1447057696) 0x555de8fdba00 con 0x555de8f26a80 -56> 2016-08-24 11:28:54.775265 7f6ce9daa700 1 -- 3.193.148.63:6800/22434 <== osd.3 3.193.149.245:6800/7279 3 ==== osd_op_reply(7 101.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v7 ==== 132+0+0 (2484205436 0 0) 0x555de8fdba00 con 0x555de8f26a80 -55> 2016-08-24 11:28:54.775315 7f6ceb8b0700 2 mds.1.1138276 boot_start 2: replaying mds log -54> 2016-08-24 11:28:54.775418 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138276:9 6.8aa3d078 201.000000c8 [read 3031877~1162427 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fda680 con 0x555de8f26a80 -53> 2016-08-24 11:28:54.775500 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.154:6800/1414 -- osd_op(mds.1.1138276:10 6.fba23765 201.000000c9 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdbd40 con 0x555de8f26f00 -52> 2016-08-24 11:28:54.775540 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138276:11 6.25f51ba0 201.000000ca [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdca40 con 0x555de8f27800 -51> 2016-08-24 11:28:54.775691 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.154:6804/1537 -- osd_op(mds.1.1138276:12 6.aafb22f0 201.000000cb [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdb6c0 con 0x555de8f28280 -50> 2016-08-24 11:28:54.775733 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138276:13 6.4dc0c2b3 201.000000cc [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdca40 con 0x555de8f27800 -49> 2016-08-24 11:28:54.775836 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138276:14 6.587f473b 201.000000cd [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdb040 con 0x555de8f28400 -48> 2016-08-24 11:28:54.775876 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138276:15 6.531b5b90 201.000000ce [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdbd40 con 0x555de8f28400 -47> 2016-08-24 11:28:54.775957 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138276:16 6.766afc26 201.000000cf [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdcd80 con 0x555de8f28580 -46> 2016-08-24 11:28:54.775996 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138276:17 6.2622d598 201.000000d0 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdd0c0 con 0x555de8f28580 -45> 2016-08-24 11:28:54.776025 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138276:18 6.aa5d77d8 201.000000d1 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdd400 con 0x555de8f28400 -44> 2016-08-24 11:28:54.776059 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138276:19 6.fff6b84b 201.000000d2 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdd740 con 0x555de8f28580 -43> 2016-08-24 11:28:54.891839 7f6ce9ba8700 1 -- 3.193.148.63:6800/22434 <== osd.8 3.193.149.154:6800/1414 2 ==== osd_op_reply(10 201.000000c9 [read 0~4194304 [fadvise_dontneed]] v0'0 uv4038 ondisk = 0) v7 ==== 132+0+4194304 (1802623356 0 3850264227) 0x555de8fdb380 con 0x555de8f26f00 -42> 2016-08-24 11:28:54.906534 7f6ce97a4700 1 -- 3.193.148.63:6800/22434 <== osd.2 3.193.150.32:6800/23404 2 ==== osd_op_reply(11 201.000000ca [read 0~4194304 [fadvise_dontneed]] v0'0 uv4103 ondisk = 0) v7 ==== 132+0+4194304 (26054290 0 1896804775) 0x555de8fdc3c0 con 0x555de8f27800 -41> 2016-08-24 11:28:54.956549 7f6ce8a9e700 1 -- 3.193.148.63:6800/22434 <== osd.9 3.193.149.154:6816/4776 1 ==== osd_op_reply(16 201.000000cf [read 0~4194304 [fadvise_dontneed]] v0'0 uv650 ondisk = 0) v7 ==== 132+0+4194304 (4242794416 0 1848106859) 0x555de8fdcd80 con 0x555de8f28580 -40> 2016-08-24 11:28:54.988659 7f6ce9daa700 1 -- 3.193.148.63:6800/22434 <== osd.3 3.193.149.245:6800/7279 4 ==== osd_op_reply(9 201.000000c8 [read 3031877~1162427 [fadvise_dontneed]] v0'0 uv5531 ondisk = 0) v7 ==== 132+0+1162427 (4161187428 0 922796555) 0x555de8fdba00 con 0x555de8f26a80 -39> 2016-08-24 11:28:54.995298 7f6ce97a4700 1 -- 3.193.148.63:6800/22434 <== osd.2 3.193.150.32:6800/23404 3 ==== osd_op_reply(13 201.000000cc [read 0~4194304 [fadvise_dontneed]] v0'0 uv5539 ondisk = 0) v7 ==== 132+0+4194304 (1566533131 0 3567643032) 0x555de8fdc3c0 con 0x555de8f27800 -38> 2016-08-24 11:28:55.016706 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138276:20 6.d5d535bb 201.000000d3 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdda80 con 0x555de8f28400 -37> 2016-08-24 11:28:55.021476 7f6ce8a9e700 1 -- 3.193.148.63:6800/22434 <== osd.9 3.193.149.154:6816/4776 2 ==== osd_op_reply(17 201.000000d0 [read 0~4194304 [fadvise_dontneed]] v0'0 uv381 ondisk = 0) v7 ==== 132+0+4194304 (2336918684 0 3938895327) 0x555de8fdcd80 con 0x555de8f28580 -36> 2016-08-24 11:28:55.070482 7f6ce8a9e700 1 -- 3.193.148.63:6800/22434 <== osd.9 3.193.149.154:6816/4776 3 ==== osd_op_reply(19 201.000000d2 [read 0~4194304 [fadvise_dontneed]] v0'0 uv4019 ondisk = 0) v7 ==== 132+0+4194304 (323544535 0 4141201163) 0x555de8fdcd80 con 0x555de8f28580 -35> 2016-08-24 11:28:55.094744 7f6ce96a3700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6801/1173 -- osd_op(mds.1.1138276:21 6.b16d511 201.000000d4 [read 0~1652680 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632278) v7 -- ?+0 0x555de8fdd0c0 con 0x555de8f26d80 -34> 2016-08-24 11:28:55.144467 7f6ce98a5700 1 -- 3.193.148.63:6800/22434 <== osd.0 3.193.148.63:6801/1173 2 ==== osd_op_reply(21 201.000000d4 [read 0~1652680 [fadvise_dontneed]] v0'0 uv2195 ondisk = 0) v7 ==== 132+0+1652680 (3386783210 0 1949932918) 0x555de8fdc700 con 0x555de8f26d80 -33> 2016-08-24 11:28:55.168575 7f6ce899d700 1 -- 3.193.148.63:6800/22434 <== osd.7 3.193.149.154:6804/1537 1 ==== osd_op_reply(12 201.000000cb [read 0~4194304 [fadvise_dontneed]] v0'0 uv1742 ondisk = 0) v7 ==== 132+0+4194304 (3247950381 0 3254678077) 0x555de8fdad00 con 0x555de8f28280 -32> 2016-08-24 11:28:55.217928 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7294,7290 used ino 2000000cd9b but session next is 2000000b24c -31> 2016-08-24 11:28:55.218006 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7296,7290 used ino 2000000cd9c but session next is 2000000b24c -30> 2016-08-24 11:28:55.218042 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7298,7290 used ino 2000000cd9d but session next is 2000000b24c -29> 2016-08-24 11:28:55.218071 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7300,7290 used ino 2000000cd9e but session next is 2000000b24c -28> 2016-08-24 11:28:55.218109 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7302,7290 used ino 2000000cd9f but session next is 2000000b24c -27> 2016-08-24 11:28:55.218139 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7304,7290 used ino 2000000cda0 but session next is 2000000b24c -26> 2016-08-24 11:28:55.218168 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7306,7290 used ino 2000000cda1 but session next is 2000000b24c -25> 2016-08-24 11:28:55.218197 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7308,7290 used ino 2000000cda2 but session next is 2000000b24c -24> 2016-08-24 11:28:55.218251 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7310,7290 used ino 2000000cda3 but session next is 2000000b24c -23> 2016-08-24 11:28:55.218338 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7312,7290 used ino 2000000cda4 but session next is 2000000b24c -22> 2016-08-24 11:28:55.218369 7f6ce96a3700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7314,7290 used ino 2000000cda5 but session next is 2000000b24c -21> 2016-08-24 11:28:56.652089 7f6ce8b9f700 1 -- 3.193.148.63:6800/22434 <== osd.1 3.193.151.182:6800/1753 1 ==== osd_op_reply(18 201.000000d1 [read 0~4194304 [fadvise_dontneed]] v0'0 uv5492 ondisk = 0) v7 ==== 132+0+4194304 (1700385294 0 4190685855) 0x555de8fdd0c0 con 0x555de8f28400 -20> 2016-08-24 11:28:56.957462 7f6ce8b9f700 1 -- 3.193.148.63:6800/22434 <== osd.1 3.193.151.182:6800/1753 2 ==== osd_op_reply(15 201.000000ce [read 0~4194304 [fadvise_dontneed]] v0'0 uv596 ondisk = 0) v7 ==== 132+0+4194304 (4243044719 0 45223128) 0x555de8fdd0c0 con 0x555de8f28400 -19> 2016-08-24 11:28:57.166905 7f6cee9b7700 10 monclient: tick -18> 2016-08-24 11:28:57.166924 7f6cee9b7700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2016-08-24 11:28:27.166922) -17> 2016-08-24 11:28:57.166940 7f6cee9b7700 10 log_client log_queue is 11 last_log 11 sent 0 num 11 unsent 11 sending 11 -16> 2016-08-24 11:28:57.166948 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.217942 mds.1 3.193.148.63:6800/22434 1 : cluster [WRN] replayed op client.24092763:7294,7290 used ino 2000000cd9b but session next is 2000000b24c -15> 2016-08-24 11:28:57.166964 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218010 mds.1 3.193.148.63:6800/22434 2 : cluster [WRN] replayed op client.24092763:7296,7290 used ino 2000000cd9c but session next is 2000000b24c -14> 2016-08-24 11:28:57.166970 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218045 mds.1 3.193.148.63:6800/22434 3 : cluster [WRN] replayed op client.24092763:7298,7290 used ino 2000000cd9d but session next is 2000000b24c -13> 2016-08-24 11:28:57.166974 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218075 mds.1 3.193.148.63:6800/22434 4 : cluster [WRN] replayed op client.24092763:7300,7290 used ino 2000000cd9e but session next is 2000000b24c -12> 2016-08-24 11:28:57.166979 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218113 mds.1 3.193.148.63:6800/22434 5 : cluster [WRN] replayed op client.24092763:7302,7290 used ino 2000000cd9f but session next is 2000000b24c -11> 2016-08-24 11:28:57.166983 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218142 mds.1 3.193.148.63:6800/22434 6 : cluster [WRN] replayed op client.24092763:7304,7290 used ino 2000000cda0 but session next is 2000000b24c -10> 2016-08-24 11:28:57.166992 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218171 mds.1 3.193.148.63:6800/22434 7 : cluster [WRN] replayed op client.24092763:7306,7290 used ino 2000000cda1 but session next is 2000000b24c -9> 2016-08-24 11:28:57.166997 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218201 mds.1 3.193.148.63:6800/22434 8 : cluster [WRN] replayed op client.24092763:7308,7290 used ino 2000000cda2 but session next is 2000000b24c -8> 2016-08-24 11:28:57.167001 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218254 mds.1 3.193.148.63:6800/22434 9 : cluster [WRN] replayed op client.24092763:7310,7290 used ino 2000000cda3 but session next is 2000000b24c -7> 2016-08-24 11:28:57.167006 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218342 mds.1 3.193.148.63:6800/22434 10 : cluster [WRN] replayed op client.24092763:7312,7290 used ino 2000000cda4 but session next is 2000000b24c -6> 2016-08-24 11:28:57.167010 7f6cee9b7700 10 log_client will send 2016-08-24 11:28:55.218374 mds.1 3.193.148.63:6800/22434 11 : cluster [WRN] replayed op client.24092763:7314,7290 used ino 2000000cda5 but session next is 2000000b24c -5> 2016-08-24 11:28:57.167016 7f6cee9b7700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -4> 2016-08-24 11:28:57.167033 7f6cee9b7700 1 -- 3.193.148.63:6800/22434 --> 3.193.148.63:6789/0 -- log(11 entries from seq 1 at 2016-08-24 11:28:55.217942) v1 -- ?+0 0x555de8f0c300 con 0x555de8f26300 -3> 2016-08-24 11:28:57.167073 7f6ced9b5700 2 -- 3.193.148.63:6800/22434 >> 3.193.148.63:6789/0 pipe(0x555de8fbc000 sd=8 :33264 s=2 pgs=6959244 cs=1 l=1 c=0x555de8f26300).reader got KEEPALIVE_ACK -2> 2016-08-24 11:28:57.815202 7f6ce8b9f700 1 -- 3.193.148.63:6800/22434 <== osd.1 3.193.151.182:6800/1753 3 ==== osd_op_reply(14 201.000000cd [read 0~4194304 [fadvise_dontneed]] v0'0 uv6284 ondisk = 0) v7 ==== 132+0+4194304 (451344706 0 1824143700) 0x555de8fdd0c0 con 0x555de8f28400 -1> 2016-08-24 11:28:58.020160 7f6ce8b9f700 1 -- 3.193.148.63:6800/22434 <== osd.1 3.193.151.182:6800/1753 4 ==== osd_op_reply(20 201.000000d3 [read 0~4194304 [fadvise_dontneed]] v0'0 uv6451 ondisk = 0) v7 ==== 132+0+4194304 (1684842316 0 2327309785) 0x555de8fdd0c0 con 0x555de8f28400 0> 2016-08-24 11:28:58.159406 7f6ce96a3700 -1 mds/journal.cc: In function 'virtual void EImportStart::replay(MDSRank*)' thread 7f6ce96a3700 time 2016-08-24 11:28:58.158246 mds/journal.cc: 2929: FAILED assert(mds->sessionmap.get_version() == cmapv) ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x555dddff22a0] 2: (EImportStart::replay(MDSRank*)+0x9e5) [0x555dddecbe75] 3: (MDLog::_replay_thread()+0xe73) [0x555ddde4ab23] 4: (MDLog::ReplayThread::entry()+0xd) [0x555dddbf584d] 5: (()+0x75ca) [0x7f6cf56a15ca] 6: (clone()+0x6d) [0x7f6cf40e1f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 kinetic 1/ 5 fuse -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.switcheroo.log --- end dump of recent events --- 2016-08-24 11:28:58.162157 7f6ce96a3700 -1 *** Caught signal (Aborted) ** in thread 7f6ce96a3700 thread_name:md_log_replay ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (()+0x514a2e) [0x555dddee4a2e] 2: (()+0x10c30) [0x7f6cf56aac30] 3: (gsignal()+0x35) [0x7f6cf40136f5] 4: (abort()+0x16a) [0x7f6cf40152fa] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x555dddff248b] 6: (EImportStart::replay(MDSRank*)+0x9e5) [0x555dddecbe75] 7: (MDLog::_replay_thread()+0xe73) [0x555ddde4ab23] 8: (MDLog::ReplayThread::entry()+0xd) [0x555dddbf584d] 9: (()+0x75ca) [0x7f6cf56a15ca] 10: (clone()+0x6d) [0x7f6cf40e1f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2016-08-24 11:28:58.162157 7f6ce96a3700 -1 *** Caught signal (Aborted) ** in thread 7f6ce96a3700 thread_name:md_log_replay ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (()+0x514a2e) [0x555dddee4a2e] 2: (()+0x10c30) [0x7f6cf56aac30] 3: (gsignal()+0x35) [0x7f6cf40136f5] 4: (abort()+0x16a) [0x7f6cf40152fa] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x555dddff248b] 6: (EImportStart::replay(MDSRank*)+0x9e5) [0x555dddecbe75] 7: (MDLog::_replay_thread()+0xe73) [0x555ddde4ab23] 8: (MDLog::ReplayThread::entry()+0xd) [0x555dddbf584d] 9: (()+0x75ca) [0x7f6cf56a15ca] 10: (clone()+0x6d) [0x7f6cf40e1f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 kinetic 1/ 5 fuse -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.switcheroo.log --- end dump of recent events --- 2016-08-24 11:28:58.414389 7f06f4841200 0 set uid:gid to 167:167 (ceph:ceph) 2016-08-24 11:28:58.414437 7f06f4841200 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mds, pid 22506 2016-08-24 11:28:58.415135 7f06f4841200 0 pidfile_write: ignore empty --pid-file 2016-08-24 11:29:04.598950 7f06ee769700 1 mds.switcheroo handle_mds_map standby 2016-08-24 11:29:05.174202 7f06ee769700 1 mds.1.1138278 handle_mds_map i am now mds.1.1138278 2016-08-24 11:29:05.174207 7f06ee769700 1 mds.1.1138278 handle_mds_map state change up:boot --> up:replay 2016-08-24 11:29:05.174217 7f06ee769700 1 mds.1.1138278 replay_start 2016-08-24 11:29:05.174220 7f06ee769700 1 mds.1.1138278 recovery set is 0 2016-08-24 11:29:05.174225 7f06ee769700 1 mds.1.1138278 waiting for osdmap 632280 (which blacklists prior instance) 2016-08-24 11:29:05.192362 7f06e945c700 0 mds.1.cache creating system inode with ino:101 2016-08-24 11:29:05.192478 7f06e945c700 0 mds.1.cache creating system inode with ino:1 2016-08-24 11:29:05.689977 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7294,7290 used ino 2000000cd9b but session next is 2000000b24c 2016-08-24 11:29:05.690056 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7296,7290 used ino 2000000cd9c but session next is 2000000b24c 2016-08-24 11:29:05.690092 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7298,7290 used ino 2000000cd9d but session next is 2000000b24c 2016-08-24 11:29:05.690121 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7300,7290 used ino 2000000cd9e but session next is 2000000b24c 2016-08-24 11:29:05.690151 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7302,7290 used ino 2000000cd9f but session next is 2000000b24c 2016-08-24 11:29:05.690187 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7304,7290 used ino 2000000cda0 but session next is 2000000b24c 2016-08-24 11:29:05.690217 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7306,7290 used ino 2000000cda1 but session next is 2000000b24c 2016-08-24 11:29:05.690251 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7308,7290 used ino 2000000cda2 but session next is 2000000b24c 2016-08-24 11:29:05.690291 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7310,7290 used ino 2000000cda3 but session next is 2000000b24c 2016-08-24 11:29:05.690376 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7312,7290 used ino 2000000cda4 but session next is 2000000b24c 2016-08-24 11:29:05.690407 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7314,7290 used ino 2000000cda5 but session next is 2000000b24c 2016-08-24 11:29:07.626310 7f06e7c52700 -1 mds/journal.cc: In function 'virtual void EImportStart::replay(MDSRank*)' thread 7f06e7c52700 time 2016-08-24 11:29:07.623086 mds/journal.cc: 2929: FAILED assert(mds->sessionmap.get_version() == cmapv) ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55af5a3782a0] 2: (EImportStart::replay(MDSRank*)+0x9e5) [0x55af5a251e75] 3: (MDLog::_replay_thread()+0xe73) [0x55af5a1d0b23] 4: (MDLog::ReplayThread::entry()+0xd) [0x55af59f7b84d] 5: (()+0x75ca) [0x7f06f3c505ca] 6: (clone()+0x6d) [0x7f06f2690f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -197> 2016-08-24 11:28:58.406289 7f06f4841200 5 asok(0x55af6461c000) register_command perfcounters_dump hook 0x55af64600030 -196> 2016-08-24 11:28:58.406308 7f06f4841200 5 asok(0x55af6461c000) register_command 1 hook 0x55af64600030 -195> 2016-08-24 11:28:58.406312 7f06f4841200 5 asok(0x55af6461c000) register_command perf dump hook 0x55af64600030 -194> 2016-08-24 11:28:58.406314 7f06f4841200 5 asok(0x55af6461c000) register_command perfcounters_schema hook 0x55af64600030 -193> 2016-08-24 11:28:58.406319 7f06f4841200 5 asok(0x55af6461c000) register_command 2 hook 0x55af64600030 -192> 2016-08-24 11:28:58.406324 7f06f4841200 5 asok(0x55af6461c000) register_command perf schema hook 0x55af64600030 -191> 2016-08-24 11:28:58.406329 7f06f4841200 5 asok(0x55af6461c000) register_command perf reset hook 0x55af64600030 -190> 2016-08-24 11:28:58.406333 7f06f4841200 5 asok(0x55af6461c000) register_command config show hook 0x55af64600030 -189> 2016-08-24 11:28:58.406338 7f06f4841200 5 asok(0x55af6461c000) register_command config set hook 0x55af64600030 -188> 2016-08-24 11:28:58.406342 7f06f4841200 5 asok(0x55af6461c000) register_command config get hook 0x55af64600030 -187> 2016-08-24 11:28:58.406347 7f06f4841200 5 asok(0x55af6461c000) register_command config diff hook 0x55af64600030 -186> 2016-08-24 11:28:58.406351 7f06f4841200 5 asok(0x55af6461c000) register_command log flush hook 0x55af64600030 -185> 2016-08-24 11:28:58.406356 7f06f4841200 5 asok(0x55af6461c000) register_command log dump hook 0x55af64600030 -184> 2016-08-24 11:28:58.406360 7f06f4841200 5 asok(0x55af6461c000) register_command log reopen hook 0x55af64600030 -183> 2016-08-24 11:28:58.414389 7f06f4841200 0 set uid:gid to 167:167 (ceph:ceph) -182> 2016-08-24 11:28:58.414437 7f06f4841200 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mds, pid 22506 -181> 2016-08-24 11:28:58.415113 7f06f4841200 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/22506 need_addr=1 -180> 2016-08-24 11:28:58.415135 7f06f4841200 0 pidfile_write: ignore empty --pid-file -179> 2016-08-24 11:28:58.417067 7f06f4841200 5 asok(0x55af6461c000) init /var/run/ceph/ceph-mds.switcheroo.asok -178> 2016-08-24 11:28:58.417092 7f06f4841200 5 asok(0x55af6461c000) bind_and_listen /var/run/ceph/ceph-mds.switcheroo.asok -177> 2016-08-24 11:28:58.417303 7f06f4841200 5 asok(0x55af6461c000) register_command 0 hook 0x55af645fe0a8 -176> 2016-08-24 11:28:58.417318 7f06f4841200 5 asok(0x55af6461c000) register_command version hook 0x55af645fe0a8 -175> 2016-08-24 11:28:58.417329 7f06f4841200 5 asok(0x55af6461c000) register_command git_version hook 0x55af645fe0a8 -174> 2016-08-24 11:28:58.417340 7f06f4841200 5 asok(0x55af6461c000) register_command help hook 0x55af64600130 -173> 2016-08-24 11:28:58.417351 7f06f4841200 5 asok(0x55af6461c000) register_command get_command_descriptions hook 0x55af64600140 -172> 2016-08-24 11:28:58.417395 7f06f4841200 10 monclient(hunting): build_initial_monmap -171> 2016-08-24 11:28:58.417397 7f06eff6c700 5 asok(0x55af6461c000) entry start -170> 2016-08-24 11:28:58.417426 7f06f4841200 1 -- 0.0.0.0:6800/22506 messenger.start -169> 2016-08-24 11:28:58.417474 7f06f4841200 5 adding auth protocol: cephx -168> 2016-08-24 11:28:58.417486 7f06f4841200 5 adding auth protocol: cephx -167> 2016-08-24 11:28:58.417550 7f06f4841200 5 asok(0x55af6461c000) register_command objecter_requests hook 0x55af64600150 -166> 2016-08-24 11:28:58.417620 7f06f4841200 1 accepter.accepter.start -165> 2016-08-24 11:28:58.417652 7f06f4841200 10 monclient(hunting): init -164> 2016-08-24 11:28:58.417664 7f06f4841200 5 adding auth protocol: cephx -163> 2016-08-24 11:28:58.417673 7f06f4841200 10 monclient(hunting): auth_supported 2 method cephx -162> 2016-08-24 11:28:58.417851 7f06f4841200 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-switcheroo/keyring -161> 2016-08-24 11:28:58.417915 7f06f4841200 10 monclient(hunting): _reopen_session rank -1 name -160> 2016-08-24 11:28:58.418060 7f06f4841200 10 monclient(hunting): picked mon.noname-a con 0x55af64650300 addr 3.193.148.63:6789/0 -159> 2016-08-24 11:28:58.418079 7f06f4841200 10 monclient(hunting): _send_mon_message to mon.noname-a at 3.193.148.63:6789/0 -158> 2016-08-24 11:28:58.418091 7f06f4841200 1 -- 0.0.0.0:6800/22506 --> 3.193.148.63:6789/0 -- auth(proto 0 35 bytes epoch 0) v1 -- ?+0 0x55af64634a00 con 0x55af64650300 -157> 2016-08-24 11:28:58.418112 7f06f4841200 10 monclient(hunting): renew_subs -156> 2016-08-24 11:28:58.420510 7f06f483b700 1 -- 3.193.148.63:6800/22506 learned my addr 3.193.148.63:6800/22506 -155> 2016-08-24 11:28:58.422648 7f06ebf64700 2 -- 3.193.148.63:6800/22506 >> 3.193.148.63:6789/0 pipe(0x55af646e6000 sd=8 :33282 s=2 pgs=6959246 cs=1 l=1 c=0x55af64650300).reader got KEEPALIVE_ACK -154> 2016-08-24 11:28:58.422710 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 1 ==== mon_map magic: 0 v1 ==== 503+0+0 (716945685 0 0) 0x55af64618900 con 0x55af64650300 -153> 2016-08-24 11:28:58.422723 7f06ee769700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 -152> 2016-08-24 11:28:58.422732 7f06ee769700 10 monclient(hunting): got monmap 5, mon.noname-a is now rank -1 -151> 2016-08-24 11:28:58.422734 7f06ee769700 10 monclient(hunting): dump: epoch 5 fsid 40216c8a-ba3c-4cec-9e80-212396e214a1 last_changed 2016-05-05 10:18:29.089450 created 0.000000 0: 3.193.148.63:6789/0 mon.switcheroo 1: 3.193.149.245:6789/0 mon.microstation 2: 3.193.150.32:6789/0 mon.dashboardpc -150> 2016-08-24 11:28:58.422777 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (196935591 0 0) 0x55af64635180 con 0x55af64650300 -149> 2016-08-24 11:28:58.422794 7f06ee769700 10 monclient(hunting): my global_id is 24534072 -148> 2016-08-24 11:28:58.422859 7f06ee769700 10 monclient(hunting): _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -147> 2016-08-24 11:28:58.422864 7f06ee769700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x55af64634f00 con 0x55af64650300 -146> 2016-08-24 11:28:58.423224 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (474653755 0 0) 0x55af64635400 con 0x55af64650300 -145> 2016-08-24 11:28:58.423320 7f06ee769700 10 monclient(hunting): _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -144> 2016-08-24 11:28:58.423325 7f06ee769700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0 0x55af64635180 con 0x55af64650300 -143> 2016-08-24 11:28:58.423772 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (3715916095 0 0) 0x55af64634a00 con 0x55af64650300 -142> 2016-08-24 11:28:58.423872 7f06ee769700 1 monclient(hunting): found mon.switcheroo -141> 2016-08-24 11:28:58.423874 7f06ee769700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -140> 2016-08-24 11:28:58.423877 7f06ee769700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x55af6460d000 con 0x55af64650300 -139> 2016-08-24 11:28:58.423916 7f06ee769700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2016-08-24 11:28:28.423915) -138> 2016-08-24 11:28:58.423925 7f06ee769700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -137> 2016-08-24 11:28:58.423929 7f06ee769700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x55af64635400 con 0x55af64650300 -136> 2016-08-24 11:28:58.423943 7f06f4841200 5 monclient: authenticate success, global_id 24534072 -135> 2016-08-24 11:28:58.423955 7f06f4841200 10 monclient: wait_auth_rotating waiting (until 2016-08-24 11:29:28.423954) -134> 2016-08-24 11:28:58.424137 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 5 ==== mon_map magic: 0 v1 ==== 503+0+0 (716945685 0 0) 0x55af64618b40 con 0x55af64650300 -133> 2016-08-24 11:28:58.424148 7f06ee769700 10 monclient: handle_monmap mon_map magic: 0 v1 -132> 2016-08-24 11:28:58.424154 7f06ee769700 10 monclient: got monmap 5, mon.switcheroo is now rank 0 -131> 2016-08-24 11:28:58.424156 7f06ee769700 10 monclient: dump: epoch 5 fsid 40216c8a-ba3c-4cec-9e80-212396e214a1 last_changed 2016-05-05 10:18:29.089450 created 0.000000 0: 3.193.148.63:6789/0 mon.switcheroo 1: 3.193.149.245:6789/0 mon.microstation 2: 3.193.150.32:6789/0 mon.dashboardpc -130> 2016-08-24 11:28:58.424380 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (1230641808 0 0) 0x55af64635680 con 0x55af64650300 -129> 2016-08-24 11:28:58.424457 7f06ee769700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2016-08-24 11:28:28.424456) -128> 2016-08-24 11:28:58.424472 7f06f4841200 10 monclient: wait_auth_rotating done -127> 2016-08-24 11:28:58.424514 7f06f4841200 10 monclient: renew_subs -126> 2016-08-24 11:28:58.424524 7f06f4841200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -125> 2016-08-24 11:28:58.424536 7f06f4841200 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x55af6460d200 con 0x55af64650300 -124> 2016-08-24 11:28:58.424559 7f06f4841200 10 monclient: renew_subs -123> 2016-08-24 11:28:58.424570 7f06f4841200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -122> 2016-08-24 11:28:58.424581 7f06f4841200 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- mon_subscribe({mdsmap=0+}) v2 -- ?+0 0x55af6460d400 con 0x55af64650300 -121> 2016-08-24 11:28:58.426743 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 7 ==== osd_map(632278..632278 src has 631672..632278) v3 ==== 32694+0+0 (649812258 0 0) 0x55af64634a00 con 0x55af64650300 -120> 2016-08-24 11:28:58.426957 7f06f4841200 5 asok(0x55af6461c000) register_command status hook 0x55af64600190 -119> 2016-08-24 11:28:58.426973 7f06f4841200 5 asok(0x55af6461c000) register_command dump_ops_in_flight hook 0x55af64600190 -118> 2016-08-24 11:28:58.426985 7f06f4841200 5 asok(0x55af6461c000) register_command ops hook 0x55af64600190 -117> 2016-08-24 11:28:58.426996 7f06f4841200 5 asok(0x55af6461c000) register_command dump_blocked_ops hook 0x55af64600190 -116> 2016-08-24 11:28:58.427007 7f06f4841200 5 asok(0x55af6461c000) register_command dump_historic_ops hook 0x55af64600190 -115> 2016-08-24 11:28:58.427019 7f06f4841200 5 asok(0x55af6461c000) register_command scrub_path hook 0x55af64600190 -114> 2016-08-24 11:28:58.427031 7f06f4841200 5 asok(0x55af6461c000) register_command tag path hook 0x55af64600190 -113> 2016-08-24 11:28:58.427042 7f06f4841200 5 asok(0x55af6461c000) register_command flush_path hook 0x55af64600190 -112> 2016-08-24 11:28:58.427053 7f06f4841200 5 asok(0x55af6461c000) register_command export dir hook 0x55af64600190 -111> 2016-08-24 11:28:58.427064 7f06f4841200 5 asok(0x55af6461c000) register_command dump cache hook 0x55af64600190 -110> 2016-08-24 11:28:58.427075 7f06f4841200 5 asok(0x55af6461c000) register_command session evict hook 0x55af64600190 -109> 2016-08-24 11:28:58.427086 7f06f4841200 5 asok(0x55af6461c000) register_command osdmap barrier hook 0x55af64600190 -108> 2016-08-24 11:28:58.427098 7f06f4841200 5 asok(0x55af6461c000) register_command session ls hook 0x55af64600190 -107> 2016-08-24 11:28:58.427109 7f06f4841200 5 asok(0x55af6461c000) register_command flush journal hook 0x55af64600190 -106> 2016-08-24 11:28:58.427120 7f06f4841200 5 asok(0x55af6461c000) register_command force_readonly hook 0x55af64600190 -105> 2016-08-24 11:28:58.427131 7f06f4841200 5 asok(0x55af6461c000) register_command get subtrees hook 0x55af64600190 -104> 2016-08-24 11:28:58.427143 7f06f4841200 5 asok(0x55af6461c000) register_command dirfrag split hook 0x55af64600190 -103> 2016-08-24 11:28:58.427154 7f06f4841200 5 asok(0x55af6461c000) register_command dirfrag merge hook 0x55af64600190 -102> 2016-08-24 11:28:58.427165 7f06f4841200 5 asok(0x55af6461c000) register_command dirfrag ls hook 0x55af64600190 -101> 2016-08-24 11:28:58.467186 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 8 ==== mdsmap(e 1138276) v1 ==== 462+0+0 (1311678917 0 0) 0x55af64619680 con 0x55af64650300 -100> 2016-08-24 11:28:58.515279 7f06f4841200 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -99> 2016-08-24 11:28:58.515308 7f06f4841200 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- mdsbeacon(24534072/switcheroo up:boot seq 1 v0) v7 -- ?+0 0x55af64710000 con 0x55af64650300 -98> 2016-08-24 11:28:58.515372 7f06ee769700 5 mds.switcheroo handle_mds_map epoch 1138276 from mon.0 -97> 2016-08-24 11:29:01.418003 7f06ecf66700 10 monclient: tick -96> 2016-08-24 11:29:01.418030 7f06ecf66700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2016-08-24 11:28:31.418026) -95> 2016-08-24 11:29:01.418222 7f06ebf64700 2 -- 3.193.148.63:6800/22506 >> 3.193.148.63:6789/0 pipe(0x55af646e6000 sd=8 :33282 s=2 pgs=6959246 cs=1 l=1 c=0x55af64650300).reader got KEEPALIVE_ACK -94> 2016-08-24 11:29:02.470435 7f06eb662700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -93> 2016-08-24 11:29:02.470472 7f06eb662700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- mdsbeacon(24534072/switcheroo up:boot seq 2 v1138276) v7 -- ?+0 0x55af64710380 con 0x55af64650300 -92> 2016-08-24 11:29:04.598881 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 9 ==== mdsmap(e 1138277) v1 ==== 687+0+0 (3214408184 0 0) 0x55af646198c0 con 0x55af64650300 -91> 2016-08-24 11:29:04.598908 7f06ee769700 5 mds.switcheroo handle_mds_map epoch 1138277 from mon.0 -90> 2016-08-24 11:29:04.598950 7f06ee769700 1 mds.switcheroo handle_mds_map standby -89> 2016-08-24 11:29:04.598963 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 10 ==== mdsbeacon(24534072/switcheroo up:boot seq 2 v1138277) v7 ==== 139+0+0 (1222240528 0 0) 0x55af64710000 con 0x55af64650300 -88> 2016-08-24 11:29:05.173891 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 11 ==== mdsmap(e 1138278) v1 ==== 968+0+0 (407388913 0 0) 0x55af64619b00 con 0x55af64650300 -87> 2016-08-24 11:29:05.173915 7f06ee769700 5 mds.switcheroo handle_mds_map epoch 1138278 from mon.0 -86> 2016-08-24 11:29:05.174034 7f06ee769700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -85> 2016-08-24 11:29:05.174077 7f06ee769700 10 monclient: renew_subs -84> 2016-08-24 11:29:05.174085 7f06ee769700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -83> 2016-08-24 11:29:05.174094 7f06ee769700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=632279}) v2 -- ?+0 0x55af6460d000 con 0x55af64650300 -82> 2016-08-24 11:29:05.174202 7f06ee769700 1 mds.1.1138278 handle_mds_map i am now mds.1.1138278 -81> 2016-08-24 11:29:05.174207 7f06ee769700 1 mds.1.1138278 handle_mds_map state change up:boot --> up:replay -80> 2016-08-24 11:29:05.174217 7f06ee769700 1 mds.1.1138278 replay_start -79> 2016-08-24 11:29:05.174220 7f06ee769700 1 mds.1.1138278 recovery set is 0 -78> 2016-08-24 11:29:05.174225 7f06ee769700 1 mds.1.1138278 waiting for osdmap 632280 (which blacklists prior instance) -77> 2016-08-24 11:29:05.175695 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 12 ==== osd_map(632279..632280 src has 631672..632280) v3 ==== 676+0+0 (2534197185 0 0) 0x55af64634f00 con 0x55af64650300 -76> 2016-08-24 11:29:05.175756 7f06ee769700 10 monclient: renew_subs -75> 2016-08-24 11:29:05.175762 7f06ee769700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -74> 2016-08-24 11:29:05.175766 7f06ee769700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- mon_subscribe({osdmap=632281}) v2 -- ?+0 0x55af6460d200 con 0x55af64650300 -73> 2016-08-24 11:29:05.175826 7f06e9e5f700 2 mds.1.1138278 boot_start 0: opening inotable -72> 2016-08-24 11:29:05.175941 7f06e9e5f700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138278:1 6.ad6e75db mds1_inotable [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ea000 con 0x55af64650f00 -71> 2016-08-24 11:29:05.175980 7f06e9e5f700 2 mds.1.1138278 boot_start 0: opening sessionmap -70> 2016-08-24 11:29:05.176040 7f06e9e5f700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.154:6812/2220 -- osd_op(mds.1.1138278:2 6.32a2fe43 mds1_sessionmap [omap-get-header 0~0,omap-get-vals 0~16] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ea340 con 0x55af64651080 -69> 2016-08-24 11:29:05.176063 7f06e9e5f700 2 mds.1.1138278 boot_start 0: opening mds log -68> 2016-08-24 11:29:05.176073 7f06e9e5f700 5 mds.1.log open discovering log bounds -67> 2016-08-24 11:29:05.176150 7f06e945c700 4 mds.1.journalpointer Reading journal pointer '401.00000000' -66> 2016-08-24 11:29:05.176223 7f06e945c700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.154:6800/1414 -- osd_op(mds.1.1138278:3 6.fcd6c765 401.00000000 [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ead00 con 0x55af64651380 -65> 2016-08-24 11:29:05.180283 7f06e8258700 1 -- 3.193.148.63:6800/22506 <== osd.3 3.193.149.245:6800/7279 1 ==== osd_op_reply(1 mds1_inotable [read 0~98] v0'0 uv3885 ondisk = 0) v7 ==== 133+0+98 (2435275223 0 3676677995) 0x55af646eb040 con 0x55af64650f00 -64> 2016-08-24 11:29:05.180296 7f06e8359700 1 -- 3.193.148.63:6800/22506 <== osd.6 3.193.149.154:6812/2220 1 ==== osd_op_reply(2 mds1_sessionmap [omap-get-header 0~0,omap-get-vals 0~16] v0'0 uv2124 ondisk = 0) v7 ==== 177+0+807 (3723646604 0 2854467212) 0x55af646eb380 con 0x55af64651080 -63> 2016-08-24 11:29:05.180409 7f06e8157700 1 -- 3.193.148.63:6800/22506 <== osd.8 3.193.149.154:6800/1414 1 ==== osd_op_reply(3 401.00000000 [read 0~22] v0'0 uv1 ondisk = 0) v7 ==== 132+0+22 (1981399345 0 369460326) 0x55af646eb6c0 con 0x55af64651380 -62> 2016-08-24 11:29:05.180446 7f06e945c700 1 mds.1.journaler(ro) recover start -61> 2016-08-24 11:29:05.180457 7f06e945c700 1 mds.1.journaler(ro) read_head -60> 2016-08-24 11:29:05.180486 7f06e945c700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.154:6812/2220 -- osd_op(mds.1.1138278:4 6.5664a157 201.00000000 [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646eba00 con 0x55af64651080 -59> 2016-08-24 11:29:05.180519 7f06e945c700 4 mds.1.log Waiting for journal 201 to recover... -58> 2016-08-24 11:29:05.184267 7f06e8359700 1 -- 3.193.148.63:6800/22506 <== osd.6 3.193.149.154:6812/2220 2 ==== osd_op_reply(4 201.00000000 [read 0~90] v0'0 uv20080 ondisk = 0) v7 ==== 132+0+90 (2995199359 0 3155850792) 0x55af646eb380 con 0x55af64651080 -57> 2016-08-24 11:29:05.186241 7f06e9e5f700 1 mds.1.journaler(ro) _finish_read_head loghead(trim 838860800, expire 841892677, write 890374656, stream_format 1). probing for end of log (from 890374656)... -56> 2016-08-24 11:29:05.186254 7f06e9e5f700 1 mds.1.journaler(ro) probing for end of the log -55> 2016-08-24 11:29:05.186332 7f06e9e5f700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6801/1173 -- osd_op(mds.1.1138278:5 6.b16d511 201.000000d4 [stat] snapc 0=[] ack+read+rwordered+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ea680 con 0x55af64651200 -54> 2016-08-24 11:29:05.186398 7f06e9e5f700 1 -- 3.193.148.63:6800/22506 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138278:6 6.3071b1a0 201.000000d5 [stat] snapc 0=[] ack+read+rwordered+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ea340 con 0x55af64651c80 -53> 2016-08-24 11:29:05.189812 7f06e7e54700 1 -- 3.193.148.63:6800/22506 <== osd.0 3.193.148.63:6801/1173 1 ==== osd_op_reply(5 201.000000d4 [stat] v0'0 uv2195 ondisk = 0) v7 ==== 132+0+16 (3872180751 0 2553466074) 0x55af646ec3c0 con 0x55af64651200 -52> 2016-08-24 11:29:05.192288 7f06e7d53700 1 -- 3.193.148.63:6800/22506 <== osd.2 3.193.150.32:6800/23404 1 ==== osd_op_reply(6 201.000000d5 [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7 ==== 132+0+0 (2126641124 0 0) 0x55af646ec700 con 0x55af64651c80 -51> 2016-08-24 11:29:05.192329 7f06e9e5f700 1 mds.1.journaler(ro) _finish_probe_end write_pos = 890845128 (header had 890374656). recovered. -50> 2016-08-24 11:29:05.192343 7f06e945c700 4 mds.1.log Journal 201 recovered. -49> 2016-08-24 11:29:05.192347 7f06e945c700 4 mds.1.log Recovered journal 201 in format 1 -48> 2016-08-24 11:29:05.192351 7f06e945c700 2 mds.1.1138278 boot_start 1: loading/discovering base inodes -47> 2016-08-24 11:29:05.192362 7f06e945c700 0 mds.1.cache creating system inode with ino:101 -46> 2016-08-24 11:29:05.192434 7f06e945c700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138278:7 6.75b54883 101.00000000 [getxattr inode] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ebd40 con 0x55af64650f00 -45> 2016-08-24 11:29:05.192468 7f06e945c700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138278:8 6.2fd1f055 101.00000000.inode [read 0~0] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646eca40 con 0x55af64650f00 -44> 2016-08-24 11:29:05.192478 7f06e945c700 0 mds.1.cache creating system inode with ino:1 -43> 2016-08-24 11:29:05.195270 7f06e8258700 1 -- 3.193.148.63:6800/22506 <== osd.3 3.193.149.245:6800/7279 2 ==== osd_op_reply(8 101.00000000.inode [read 0~508] v0'0 uv42 ondisk = 0) v7 ==== 138+0+508 (2337917120 0 1447057696) 0x55af646eb040 con 0x55af64650f00 -42> 2016-08-24 11:29:05.195309 7f06e8258700 1 -- 3.193.148.63:6800/22506 <== osd.3 3.193.149.245:6800/7279 3 ==== osd_op_reply(7 101.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v7 ==== 132+0+0 (422715158 0 0) 0x55af646eb040 con 0x55af64650f00 -41> 2016-08-24 11:29:05.195349 7f06e9e5f700 2 mds.1.1138278 boot_start 2: replaying mds log -40> 2016-08-24 11:29:05.195432 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.245:6800/7279 -- osd_op(mds.1.1138278:9 6.8aa3d078 201.000000c8 [read 3031877~1162427 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ea000 con 0x55af64650f00 -39> 2016-08-24 11:29:05.195518 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.154:6800/1414 -- osd_op(mds.1.1138278:10 6.fba23765 201.000000c9 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ebd40 con 0x55af64651380 -38> 2016-08-24 11:29:05.195599 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138278:11 6.25f51ba0 201.000000ca [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ed0c0 con 0x55af64651c80 -37> 2016-08-24 11:29:05.195730 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.154:6804/1537 -- osd_op(mds.1.1138278:12 6.aafb22f0 201.000000cb [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ea340 con 0x55af64652400 -36> 2016-08-24 11:29:05.195769 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.150.32:6800/23404 -- osd_op(mds.1.1138278:13 6.4dc0c2b3 201.000000cc [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ed0c0 con 0x55af64651c80 -35> 2016-08-24 11:29:05.195886 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138278:14 6.587f473b 201.000000cd [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ead00 con 0x55af64652580 -34> 2016-08-24 11:29:05.195926 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138278:15 6.531b5b90 201.000000ce [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ebd40 con 0x55af64652580 -33> 2016-08-24 11:29:05.196009 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138278:16 6.766afc26 201.000000cf [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ecd80 con 0x55af64652700 -32> 2016-08-24 11:29:05.196047 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138278:17 6.2622d598 201.000000d0 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ed400 con 0x55af64652700 -31> 2016-08-24 11:29:05.196086 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138278:18 6.aa5d77d8 201.000000d1 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ed740 con 0x55af64652580 -30> 2016-08-24 11:29:05.196119 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.149.154:6816/4776 -- osd_op(mds.1.1138278:19 6.fff6b84b 201.000000d2 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646eda80 con 0x55af64652700 -29> 2016-08-24 11:29:05.324824 7f06e8157700 1 -- 3.193.148.63:6800/22506 <== osd.8 3.193.149.154:6800/1414 2 ==== osd_op_reply(10 201.000000c9 [read 0~4194304 [fadvise_dontneed]] v0'0 uv4038 ondisk = 0) v7 ==== 132+0+4194304 (3864146198 0 3850264227) 0x55af646eb6c0 con 0x55af64651380 -28> 2016-08-24 11:29:05.372961 7f06e7d53700 1 -- 3.193.148.63:6800/22506 <== osd.2 3.193.150.32:6800/23404 2 ==== osd_op_reply(11 201.000000ca [read 0~4194304 [fadvise_dontneed]] v0'0 uv4103 ondisk = 0) v7 ==== 132+0+4194304 (2360239864 0 1896804775) 0x55af646ec700 con 0x55af64651c80 -27> 2016-08-24 11:29:05.406480 7f06e6f4c700 1 -- 3.193.148.63:6800/22506 <== osd.9 3.193.149.154:6816/4776 1 ==== osd_op_reply(17 201.000000d0 [read 0~4194304 [fadvise_dontneed]] v0'0 uv381 ondisk = 0) v7 ==== 132+0+4194304 (107570422 0 3938895327) 0x55af646ecd80 con 0x55af64652700 -26> 2016-08-24 11:29:05.453983 7f06e7d53700 1 -- 3.193.148.63:6800/22506 <== osd.2 3.193.150.32:6800/23404 3 ==== osd_op_reply(13 201.000000cc [read 0~4194304 [fadvise_dontneed]] v0'0 uv5539 ondisk = 0) v7 ==== 132+0+4194304 (3497823841 0 3567643032) 0x55af646ec700 con 0x55af64651c80 -25> 2016-08-24 11:29:05.473838 7f06e6f4c700 1 -- 3.193.148.63:6800/22506 <== osd.9 3.193.149.154:6816/4776 2 ==== osd_op_reply(16 201.000000cf [read 0~4194304 [fadvise_dontneed]] v0'0 uv650 ondisk = 0) v7 ==== 132+0+4194304 (1908412378 0 1848106859) 0x55af646ecd80 con 0x55af64652700 -24> 2016-08-24 11:29:05.522196 7f06e8258700 1 -- 3.193.148.63:6800/22506 <== osd.3 3.193.149.245:6800/7279 4 ==== osd_op_reply(9 201.000000c8 [read 3031877~1162427 [fadvise_dontneed]] v0'0 uv5531 ondisk = 0) v7 ==== 132+0+1162427 (1965377038 0 922796555) 0x55af646eb040 con 0x55af64650f00 -23> 2016-08-24 11:29:05.541800 7f06e704d700 1 -- 3.193.148.63:6800/22506 <== osd.7 3.193.149.154:6804/1537 1 ==== osd_op_reply(12 201.000000cb [read 0~4194304 [fadvise_dontneed]] v0'0 uv1742 ondisk = 0) v7 ==== 132+0+4194304 (1286873671 0 3254678077) 0x55af664ec000 con 0x55af64652400 -22> 2016-08-24 11:29:05.545398 7f06e6f4c700 1 -- 3.193.148.63:6800/22506 <== osd.9 3.193.149.154:6816/4776 3 ==== osd_op_reply(19 201.000000d2 [read 0~4194304 [fadvise_dontneed]] v0'0 uv4019 ondisk = 0) v7 ==== 132+0+4194304 (2657815997 0 4141201163) 0x55af646ecd80 con 0x55af64652700 -21> 2016-08-24 11:29:05.550101 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.151.182:6800/1753 -- osd_op(mds.1.1138278:20 6.d5d535bb 201.000000d3 [read 0~4194304 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ed400 con 0x55af64652580 -20> 2016-08-24 11:29:05.601475 7f06e7c52700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6801/1173 -- osd_op(mds.1.1138278:21 6.b16d511 201.000000d4 [read 0~1652680 [fadvise_dontneed]] snapc 0=[] ack+read+known_if_redirected+full_force e632280) v7 -- ?+0 0x55af646ead00 con 0x55af64651200 -19> 2016-08-24 11:29:05.646262 7f06e7e54700 1 -- 3.193.148.63:6800/22506 <== osd.0 3.193.148.63:6801/1173 2 ==== osd_op_reply(21 201.000000d4 [read 0~1652680 [fadvise_dontneed]] v0'0 uv2195 ondisk = 0) v7 ==== 132+0+1652680 (1157488000 0 1949932918) 0x55af646ec3c0 con 0x55af64651200 -18> 2016-08-24 11:29:05.689977 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7294,7290 used ino 2000000cd9b but session next is 2000000b24c -17> 2016-08-24 11:29:05.690056 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7296,7290 used ino 2000000cd9c but session next is 2000000b24c -16> 2016-08-24 11:29:05.690092 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7298,7290 used ino 2000000cd9d but session next is 2000000b24c -15> 2016-08-24 11:29:05.690121 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7300,7290 used ino 2000000cd9e but session next is 2000000b24c -14> 2016-08-24 11:29:05.690151 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7302,7290 used ino 2000000cd9f but session next is 2000000b24c -13> 2016-08-24 11:29:05.690187 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7304,7290 used ino 2000000cda0 but session next is 2000000b24c -12> 2016-08-24 11:29:05.690217 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7306,7290 used ino 2000000cda1 but session next is 2000000b24c -11> 2016-08-24 11:29:05.690251 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7308,7290 used ino 2000000cda2 but session next is 2000000b24c -10> 2016-08-24 11:29:05.690291 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7310,7290 used ino 2000000cda3 but session next is 2000000b24c -9> 2016-08-24 11:29:05.690376 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7312,7290 used ino 2000000cda4 but session next is 2000000b24c -8> 2016-08-24 11:29:05.690407 7f06e7c52700 0 log_channel(cluster) log [WRN] : replayed op client.24092763:7314,7290 used ino 2000000cda5 but session next is 2000000b24c -7> 2016-08-24 11:29:06.173661 7f06e714e700 1 -- 3.193.148.63:6800/22506 <== osd.1 3.193.151.182:6800/1753 1 ==== osd_op_reply(18 201.000000d1 [read 0~4194304 [fadvise_dontneed]] v0'0 uv5492 ondisk = 0) v7 ==== 132+0+4194304 (3900324452 0 4190685855) 0x55af646ea680 con 0x55af64652580 -6> 2016-08-24 11:29:06.427423 7f06eb662700 10 monclient: _send_mon_message to mon.switcheroo at 3.193.148.63:6789/0 -5> 2016-08-24 11:29:06.427455 7f06eb662700 1 -- 3.193.148.63:6800/22506 --> 3.193.148.63:6789/0 -- mdsbeacon(24534072/switcheroo up:replay seq 3 v1138278) v7 -- ?+0 0x55af64710e00 con 0x55af64650300 -4> 2016-08-24 11:29:06.427731 7f06ee769700 1 -- 3.193.148.63:6800/22506 <== mon.0 3.193.148.63:6789/0 13 ==== mdsbeacon(24534072/switcheroo up:replay seq 3 v1138278) v7 ==== 135+0+0 (705157266 0 0) 0x55af64711500 con 0x55af64650300 -3> 2016-08-24 11:29:06.513344 7f06e714e700 1 -- 3.193.148.63:6800/22506 <== osd.1 3.193.151.182:6800/1753 2 ==== osd_op_reply(15 201.000000ce [read 0~4194304 [fadvise_dontneed]] v0'0 uv596 ondisk = 0) v7 ==== 132+0+4194304 (1908691205 0 45223128) 0x55af646ea680 con 0x55af64652580 -2> 2016-08-24 11:29:07.344349 7f06e714e700 1 -- 3.193.148.63:6800/22506 <== osd.1 3.193.151.182:6800/1753 3 ==== osd_op_reply(14 201.000000cd [read 0~4194304 [fadvise_dontneed]] v0'0 uv6284 ondisk = 0) v7 ==== 132+0+4194304 (2546274600 0 1824143700) 0x55af646ea680 con 0x55af64652580 -1> 2016-08-24 11:29:07.478896 7f06e714e700 1 -- 3.193.148.63:6800/22506 <== osd.1 3.193.151.182:6800/1753 4 ==== osd_op_reply(20 201.000000d3 [read 0~4194304 [fadvise_dontneed]] v0'0 uv6451 ondisk = 0) v7 ==== 132+0+4194304 (3914288934 0 2327309785) 0x55af646ea680 con 0x55af64652580 0> 2016-08-24 11:29:07.626310 7f06e7c52700 -1 mds/journal.cc: In function 'virtual void EImportStart::replay(MDSRank*)' thread 7f06e7c52700 time 2016-08-24 11:29:07.623086 mds/journal.cc: 2929: FAILED assert(mds->sessionmap.get_version() == cmapv) ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x80) [0x55af5a3782a0] 2: (EImportStart::replay(MDSRank*)+0x9e5) [0x55af5a251e75] 3: (MDLog::_replay_thread()+0xe73) [0x55af5a1d0b23] 4: (MDLog::ReplayThread::entry()+0xd) [0x55af59f7b84d] 5: (()+0x75ca) [0x7f06f3c505ca] 6: (clone()+0x6d) [0x7f06f2690f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 kinetic 1/ 5 fuse -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.switcheroo.log --- end dump of recent events --- 2016-08-24 11:29:07.632388 7f06e7c52700 -1 *** Caught signal (Aborted) ** in thread 7f06e7c52700 thread_name:md_log_replay ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (()+0x514a2e) [0x55af5a26aa2e] 2: (()+0x10c30) [0x7f06f3c59c30] 3: (gsignal()+0x35) [0x7f06f25c26f5] 4: (abort()+0x16a) [0x7f06f25c42fa] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55af5a37848b] 6: (EImportStart::replay(MDSRank*)+0x9e5) [0x55af5a251e75] 7: (MDLog::_replay_thread()+0xe73) [0x55af5a1d0b23] 8: (MDLog::ReplayThread::entry()+0xd) [0x55af59f7b84d] 9: (()+0x75ca) [0x7f06f3c505ca] 10: (clone()+0x6d) [0x7f06f2690f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2016-08-24 11:29:07.632388 7f06e7c52700 -1 *** Caught signal (Aborted) ** in thread 7f06e7c52700 thread_name:md_log_replay ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374) 1: (()+0x514a2e) [0x55af5a26aa2e] 2: (()+0x10c30) [0x7f06f3c59c30] 3: (gsignal()+0x35) [0x7f06f25c26f5] 4: (abort()+0x16a) [0x7f06f25c42fa] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x26b) [0x55af5a37848b] 6: (EImportStart::replay(MDSRank*)+0x9e5) [0x55af5a251e75] 7: (MDLog::_replay_thread()+0xe73) [0x55af5a1d0b23] 8: (MDLog::ReplayThread::entry()+0xd) [0x55af59f7b84d] 9: (()+0x75ca) [0x7f06f3c505ca] 10: (clone()+0x6d) [0x7f06f2690f6d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 newstore 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 kinetic 1/ 5 fuse -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.switcheroo.log --- end dump of recent events --- 2016-08-24 11:29:07.913407 7f0062bbf200 0 set uid:gid to 167:167 (ceph:ceph) 2016-08-24 11:29:07.913454 7f0062bbf200 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process ceph-mds, pid 22615 2016-08-24 11:29:07.913714 7f0062bbf200 0 pidfile_write: ignore empty --pid-file